Cooling system for electronic modules

ABSTRACT

A system and a method for cooling a plurality of electronic devices housed in a housing of an electronic module. The system comprises a first cooling circulatory arrangement, configured to circulate a first liquid coolant between a first electronic device of the plurality of electronic devices and a heat exchanger, the first electronic device being thermally coupled to the first liquid coolant such that heat is transferred from the first electronic device to the first liquid coolant;. The system further comprises a second cooling circulatory arrangement, configured to circulate a second liquid coolant between a second electronic device of the plurality of electronic devices and the heat exchanger, the second electronic device being thermally coupled to the second liquid coolant such that heat is transferred from the second electronic device to the second liquid coolant. The first cooling circulatory arrangement and the second cooling circulatory arrangement are thermally coupled at least via the heat exchanger, such that heat is transferred from the first liquid coolant to the second liquid coolant via the heat exchanger.

TECHNICAL FIELD OF THE DISCLOSURE

The disclosure concerns a system for cooling a plurality of electronicdevices housed in an electronic module. The system comprises a firstcooling circulatory arrangement which is arranged to cool a firstelectronic device within the electronic module, and a second coolingcirculatory arrangement arranged to cool a second electronic devicewithin the electronic module. Coolant circulating within the firstcooling circulatory arrangement is cooled by transfer of heat, via aheat exchanger, to coolant circulating in the second cooling circulatoryarrangement. There is further described a method for cooling a pluralityof electronic devices housed in an electronic module.

The disclosure also concerns a system for cooling an electronic moduleconfigured for installation into a rack. The disclosure furtherconsiders a method of installation of a liquid cooled system for anelectronic module, and a kit for use in the installation method.

BACKGROUND TO THE DISCLOSURE

Within computers, servers, or other devices used for data processing(referred to as IT, or Information Technology), are a number ofelectronic devices called Integrated Circuits (IC). The electronicdevices within the integrated circuits may include central processingunits (CPUs), Application Specific Integrated Circuits (ASICs),Graphical Processing Units (GPUs), Random Access Memory (RAM), etc. Eachof these devices produce heat when in use. In order to maintain thedevices at an optimum temperature for correct operation, it is importantfor this heat to be transferred away from the devices. As the processingpower of IT increases and so the number of electronic devices within acomputer, server or other IT grows, the challenge of removing sufficientheat created by the electronic devices increases.

The electronic devices, normally mounted on a printed circuit board(PCB), are usually housed or enclosed within a case or chassis, to forman electronic module. For instance, a computer server often comprises anumber of electronic modules, mounted in a rack and connected togetherin order to provide the required IT facilities. A method for removal ofheat from each case or chassis is required, in order to maintain theelectronic devices within the chassis at an appropriate temperature.

It is common to cool an electronic module by passing air over or througheach case or chassis. The flow of air may be sufficient to remove someheat from inside the enclosure, to the surrounding environment. Thismethod of cooling has, until recently, been used almost exclusively formass-manufactured IT and server equipment. However, it has been foundthat, as technology size decreases for the same performance, the heatproduced by electronic devices is increasing even as the footprintdecreases. As such, the peak performance of IT systems has beenthrottled by the limitations of cooling an electronic module with aircooled systems.

Accordingly, more complex systems and methods for cooling electronicmodules have been proposed. In some cases, liquid cooling has been used,in which a liquid coolant is flowed over, or flowed in proximity to aheat sink coupled to, the electronic devices. The heat can then betransferred away from the electronic devices, to an area or element atwhich the heat can be removed from the liquid coolant. Liquid coolingcan in some cases provide more efficient transfer of heat away from theelectronic devices or components, and so a greater cooling power thanair cooled systems. However, state of the art liquid cooling systemsoften required customised systems, which are complex and expensive toinstall.

Thus, it is an objective of the present invention to provide a systemfor cooling an electronic module, and furthermore a method of coolingsuch a system, which overcomes these drawbacks of prior art systems.

SUMMARY OF THE DISCLOSURE

Against this background there is provided a system and method forcooling a plurality of electronic devices housed in a housing of anelectronic module. In particular, the system comprises a first andsecond cooling circulatory arrangement, or first and second coolingloop, each circulating respective first and second liquid coolants. Eachof the cooling circulatory systems are used to cool electronic devicesof the plurality of devices within an electronic module. However, thesecond cooling circulatory arrangement is further configured to cool thecoolant circulating in the first cooling circulatory system, by exchangeof heat between the first and second liquid coolants via a heatexchanger. The first and second cooling circulatory arrangement may havedifferent efficiencies, and so a more efficient cooling provided by thesecond cooling circulatory arrangement can be used to cool the firstliquid coolant. Furthermore, the more efficient second coolingcirculatory arrangement can be focussed on specific high powerelectronic devices (which generate comparatively more heat). The firstliquid coolant can be used to maintain a generally lower temperature forthe overall environment within the electronic module.

In a preferred example, the second cooling circulatory arrangement cancomprise cold plates, which can be thermally coupled to specificelectronic devices of the plurality of electronic devices. Incomparison, the first cooling circulatory system can circulate a firstliquid coolant contained, in part, in a reservoir within the electronicmodule, wherein a number of the electronic devices are at leastpartially immersed in the reservoir of the first liquid coolant.Moreover, the inclusion of a weir or bathtub heat sink within the firstcooling circulatory system can provide further advantages for thedescribed system and method, as discussed below.

In a first aspect, there is described a system for cooling a pluralityof electronic devices housed in a housing of an electronic module, thesystem comprising:

a first cooling circulatory arrangement, configured to circulate a firstliquid coolant between a first electronic device of the plurality ofelectronic devices and a heat exchanger, the first electronic devicebeing thermally coupled to the first liquid coolant such that heat istransferred from the first electronic device to the first liquidcoolant; and

a second cooling circulatory arrangement, configured to circulate asecond liquid coolant between a second electronic device of theplurality of electronic devices and the heat exchanger, the secondelectronic device being thermally coupled to the second liquid coolantsuch that heat is transferred from the second electronic device to thesecond liquid coolant;

wherein the first cooling circulatory arrangement and the second coolingcirculatory arrangement are thermally coupled at least via the heatexchanger, such that heat is transferred from the first liquid coolantto the second liquid coolant via the heat exchanger.

The electronic devices may be any heat generating devices or components,including Integrated Circuits (IC) comprising central processing units(CPUs), Application Specific Integrated Circuits (ASICs), GraphicalProcessing Units (GPUs), Random Access Memory (RAM), etc. Together thedevices may be connected to form a server or other computer processingfunction, or other IT.

The electronic modules, or server modules, may be a module forming partof a computer server. The electronic module may have a chassis orhousing, in which each of the electronic devices are mounted. Theelectronic module may be configured for mounting or installation into arack. For instance, the electronic module may conform to the industrystandard dimensions required to fit into a standard server rack (knownas 1RU (one rack unit) or 1OU (one open unit). Such units may bereferred to as a blade server.

The system comprises a first and a second cooling circulatoryarrangement (or first and second cooling loop). The cooling circulatoryarrangements provide a configuration for flow of a respective first andsecond coolant through the electronic module.

Specifically, the first cooling circulatory arrangement circulates orflows a first liquid coolant from at least a first electronic, heatgenerating device (from which heat is absorbed by the first liquidcoolant) to a heat exchanger. Heat can be removed from the first liquidcoolant at the heat exchanger.

The second cooling circulatory arrangement circulates or flows a secondliquid coolant from at least a second electronic, heat generating device(from which heat is absorbed by the second liquid coolant) to the heatexchanger. At the heat exchanger, the heat from the first liquid coolantis received by the second liquid coolant.

Although in some cases the second cooling circulatory arrangement may beclosed (in other words, the liquid coolant is recirculated and recycledwithin the loop), this is not always the case. In alternative cases, thesecond cooling circulatory arrangement can describe an open loop, inwhich liquid coolant is received, flowed around the described pathway,and then passed to a drain. For instance, after passing through the heatexchanger the second liquid coolant in the second cooling circulatoryarrangement is then either cooled (by passing through a cooling systembefore passing back to the electronic module), or is replenished (forexample, where the second liquid coolant is part of a facility widecoolant supply, such as a facility water supply).

Beneficially, the described system is a hybrid system of two coolingcirculatory arrangements. The use of such a hybrid system allows ahigher performance, more efficient cooling arrangement to be targeted atparticularly high temperature components (the second cooling circulatoryarrangement, targeted to at least the second electronic device), as wellas using a further cooling arrangement to cool other devices. However,more than simply using two entirely separate cooling systems inparallel, the present inventors have recognised that the return flow ofthe higher performance cooling arrangement can also be used to removeheat from the other, lower performance cooling arrangement. To someextent, the first cooling circulatory arrangement may be considered tobe nested with the second cooling circulatory arrangement.

Optionally, the first liquid coolant is a dielectric liquid, and thesecond liquid coolant is water. It will be understood that although theterm liquid coolant is used herein, any suitable fluid coolant could beused.

Optionally, the second cooling circulatory arrangement further comprisesa cooling system, wherein the second cooling circulatory arrangement isconfigured to circulate the second liquid coolant between the secondelectronic device of the plurality of electronic devices, the heatexchanger and the cooling system, wherein heat is removed from thesecond liquid coolant by the cooling system. Preferably, the coolingsystem is arranged to be external to the electronic module. In otherwords, the second cooling circulatory arrangement forms a closed loop,in which the second liquid coolant received from the electronic moduleis cooled by a cooling system, before being returned to the electronicmodule for cooling the second electronic device.

Alternatively, the second cooling circulatory arrangement is connectedto a second liquid coolant supply, wherein the second coolingcirculatory arrangement is configured to circulate the second liquidcoolant received from the second liquid coolant supply between thesecond electronic device of the plurality of electronic devices and theheat exchanger, and to be returned to the second liquid coolant supply.In other words, the second cooling circulatory arrangement is an openloop, and the second liquid coolant is fed from a facility level supply,and constantly replenished. For instance, the second liquid coolantsupply may be a water supply, from which water is received (as thesecond liquid coolant), circulated through the second coolingcirculatory arrangement, and then allowed to exit the second coolingcirculatory arrangement to facility drainage.

Preferably, the heat exchanger comprises at least a first and a secondchamber separated by a thermal interface, wherein the heat exchanger isconfigured for flow of the first liquid coolant through at least thefirst chamber, and flow of the second liquid coolant through at leastthe second chamber, such that heat is transferred from the first liquidcoolant to the second liquid coolant through the thermal interface. Theheat exchanger is a dedicated element configured for exchange of heatbetween the first and second liquid coolant. Each of the first and thesecond liquid coolant may pass through one or more dedicated chambers ofthe heat exchanger, wherein heat can pass from the first to the secondliquid coolant via a thermal interface between the chambers. The heatexchanger may be of any suitable design, and may provide a plurality ofchambers and a plurality of thermal interfaces, in order to improve theefficiency of heat exchange. Fins or other protrusions may be providedat the thermal interface, to increase the surface area of the thermalinterface and promote efficiency of heat exchange between the first andsecond liquid coolant. Optionally, the heat exchanger is a plate heatexchanger.

Preferably, the heat exchanger is arranged within the housing of theelectronic module. In particular, the heat exchanger is contained withinthe electronic module. This allows at least the first liquid coolant tobe entirely retained within the electronic module. This reduces thecomplexity of connections into and out of the electronic module. It alsoallows provision of the electronic module as a sealed module, which maybe advantageous where the second liquid coolant is a dielectric, andwhich may be harmful to humans if released from the module, and whichmay be expensive to replace if leaked or lost.

Optionally, the first cooling circulatory arrangement is containedentirely within the housing of the electronic module. In other words,the first cooling circulatory arrangement is arranged so that the firstliquid coolant does not leave the confines of the housing of theelectronic module when in normal operation.

The first cooling circulatory arrangement may be at least partiallyinsulated from the second cooling circulatory arrangement, in theportions of the second cooling circulatory arrangement prior to thesecond liquid coolant circulating to the second electronic device,having the coolest second liquid coolant. The first cooling circulatoryarrangement may be at least partially insulated from the second coolingcirculatory arrangement, except at the heat exchanger at which the firstand second cooling circulatory arrangement are thermally coupled. Inother words, the return flow of the second cooling circulatoryarrangement (in other words, after receipt of heat from the secondelectronic device) is used to cool the first liquid coolant. This avoidsincreasing the temperature of the second liquid coolant before reachingthe second electronic device, in order to maximise the cooling power (ormore specifically, the temperature gradient) at the second electronicdevice.

Optionally, the housing of the electronic module contains the firstliquid coolant, and the first electronic device is at least partiallyimmersed in the first liquid coolant. In other words, the first coolingcirculatory arrangement is an immersion cooling arrangement. Heat may betransferred directly to the first liquid coolant from a surface of thefirst electronic device that is at least partially immersed in the firstliquid coolant. Portions of the second cooling circulatory arrangement,including the second electronic device, may also be at least partiallyimmersed in the first liquid coolant.

In contrast, the second cooling circulatory arrangement may beconfigured to circulate the second liquid coolant through a coolingmodule and the heat exchanger, where the cooling module is mounted tothe second electronic device at a mounting surface of the coolingmodule. Therefore, heat is exchanged between the second electronicdevice and the second liquid coolant indirectly, through the mountingsurface of the second cooling module. The cooling module forming part ofthe second cooling circulatory arrangement is discussed in more detailbelow.

In a preferred example, the first cooling circulatory arrangementcomprises a weir, the weir comprising:

a base and a retaining wall extending from the base, the base andretaining wall defining a volume for holding some of the first liquidcoolant;

an inlet, through which the first liquid coolant flows into the volume;

wherein the flow of sufficient first liquid coolant through the inletinto the volume causes the first liquid coolant to overflow theretaining wall and collect with first liquid coolant contained in thehousing of the electronic module and exterior the weir.

The base and retaining walls may provide a container or ‘bath tub’ fromwhich the first coolant may overflow. The weir may be coupled to thesurface of a first electronic device, in order to act as a heat sink forthe first electronic device. Thus, the weir provides a volume forholding or retaining liquid coolant against the heat-generatingelectronic device. Alternatively, or in addition, the weir may bemounted on a PCB that is raised compared to other components in theelectronic device, and/or compared to the level of first coolant withinthe cavity of the housing of the electronic module. In this way, thefirst liquid coolant then acts to flow over the first electronic deviceand a number of other electronic devices or components housed in theelectronic module, as it overflows the weir.

The weir may be configured to direct the flow of the first liquidcoolant circulating through the first cooling circulatory arrangement.In other words, the weir may be configured such that first liquidcoolant overflowing the weir flows on to or over specific electronicdevices housed within the electronic module. Advantageously, byinclusion of the weir in the first cooling circulatory arrangement theliquid coolant can be applied more effectively to the place or placeswhere the most heat is generated. Less coolant can therefore be used.Since the coolant is expensive and heavy, reducing the quantity ofcoolant can improve flexibility, efficiency and reliability (forexample, since coolant leakages are less likely and because the coolantin the volume can resist instant temperature changes caused by thefailure of other components in the system).

In respect of the weir, the volume for holding or retaining the firstliquid coolant can be defined by a base and a retaining wall (which maybe integral or separate). The base is the part of the weir which may bemounted on top of an electronic device (more specifically, aheat-transmitting surface of an electronic device) and transfers heatfrom the heat-transmitting surface. The base typically has a planarsurface defining the volume (and the base itself may be planar inshape). Heat transferred (typically conducted) through the base (inparticular its surface defining the volume) is transferred to the liquidcoolant held in the volume. The retaining wall extends from the base.

One effect of the weir is to raise the level of the coolant held withinthe weir's volume above that external the volume (at least when thecooling module is operated with the plane of the electronic deviceand/or circuit board horizontal) and the quantity of coolant within thecontainer of the cooling module lower than the height of the retainingwall.

Advantageously, the heat sink has projections (such as pins and/or fins)extending from the base (or less preferably, from the retaining wall)within the volume. The projections may cause the liquid coolant tospread in a radial direction away from a predetermined point on asurface of the base (for example coincident with a hottest part of theelectronic device). In particular, the projections may be formed in anon-linear pattern.

Preferably, the inlet to the weir further comprises a nozzlearrangement, for directing the first liquid coolant flowing into thevolume. The nozzle arrangement may comprise one or more nozzle (whichmay be push-fit), each of which directs the flowing or pumped firstliquid coolant to a respective part of the volume of the weir,particularly a part of the weir's base. The one or more nozzle may eachbe arranged in the base, in the retaining wall or arranged over the topof the volume to cause first liquid coolant to flow into the volume. Forinstance, each nozzle may direct the flowing or pumped liquid coolant toa respective part of the volume of the weir adjacent a part of aheat-transmitting surface of an electronic device having a maximumtemperature or a temperature above a threshold level (that is, one ofthe hottest parts of the device). Most preferably, the nozzlearrangement directs the flowing or pumped liquid coolant in a directionperpendicular to the base of the weir. This may force the coolantdirectly into the volume and improve heat dissipation.

Preferably, the first cooling circulatory arrangement further comprisesa pump configured to circulate the first liquid coolant around the firstcooling circulatory arrangement. The pump may be arranged to receivefirst liquid coolant from the reservoir of liquid coolant containedwithin the electronic module, and in which at least the first electronicdevice is at least partially immersed. The pump may then move thereceived first liquid coolant to another region of the electronicmodule, for instance to the heat exchanger, and then onwards to theinlet of the weir. The pump may be at least partially immersed in thefirst liquid coolant, the first liquid coolant thereby also assisting inthe cooling of the pump.

The first cooling circulatory arrangement may further comprise a pumpinlet, arranged to receive first liquid coolant contained in the housingof the electronic module and exterior the weir. In other words, firstliquid coolant contained within the electronic module may be received bythe pump inlet, to be passed to the pump.

Preferably, the first cooling circulatory arrangement further comprisesat least a first and a second pipe, arranged to transport the firstliquid coolant from the pump to the heat exchanger and the from the heatexchanger to the inlet of the weir, respectively.

Preferably, the second cooling circulatory arrangement further comprisesa cooling module configured to thermally couple the second electronicdevice to the second liquid coolant. The cooling module may be aspecific component for efficient heat transfer from the secondelectronic device to the second liquid coolant. The cooling module maybe mounted to the second electronic device via a mounting surface of thecooling module, so that heat is transferred from the second electronicdevice to the second liquid coolant through the mounting surface.

Preferably, the cooling module comprises a cold plate, the cold platecomprising:

a cold plate housing, a surface of the cold plate housing being arrangedto provide a thermal interface for cooling the second electronic devicewhich is thermally coupled thereto; and

at least one channel within the cold plate housing and proximate to thesurface of the cold plate housing, the at least one channel arranged forthe second liquid coolant to flow therethough such that heat receivedfrom the second electronic device through the surface of the cold platehousing is transferred to the second liquid coolant.

Advantageously, the cold plate provides an efficient and effectivemechanism for cooling specific electronic devices in the electronicmodule. The cold plate provides high performance cooling for the secondelectronic device to which it is thermally coupled. Therefore, the coldplate can be coupled to the hottest component or components within theelectronic module, in order to provide the greatest cooling power tothese components.

Optionally, the surface of the cold plate housing may be directlycoupled to a surface of the second electronic device. Alternatively, thehousing may be coupled by a further interfacing surface or component.Nevertheless, the cold plate and the second electronic device will bethermally coupled, to promote effective and efficient heat transfer fromthe second electronic device to the second liquid coolant.

More than one cold plate may be arranged in the electronic module, aspart of the second cooling circulatory arrangement. Two or more coldplates may be arranged in the second cooling circulatory arrangement inparallel or in series, or where three or more cold plates are used, acombination of parallel and series configurations could be implemented.

Preferably, the second cooling circulatory arrangement comprises aplurality of conduits arranged to transport the second liquid coolantbetween the cold plate and the heat exchanger, as well as for connectionto any cooling system or coolant supply that is external to theelectronic module.

In a second aspect, there is described a method for cooling a pluralityof electronic devices housed in a housing of an electronic module, themethod comprising:

circulating a first liquid coolant around a first cooling circulatoryarrangement, comprising circulating a first liquid coolant between afirst electronic device of the plurality of electronic devices and aheat exchanger, the first electronic device being thermally coupled tothe first liquid coolant such that heat is transferred from the firstelectronic device to the first liquid coolant; and

circulating a second liquid coolant around a second cooling circulatoryarrangement, comprising circulating a second liquid coolant between asecond electronic device of the plurality of electronic devices and theheat exchanger, the second electronic device being thermally coupled tothe second liquid coolant such that heat is transferred from the secondelectronic device to the second liquid coolant;

wherein the first cooling circulatory arrangement and the second coolingcirculatory arrangement are thermally coupled at least via the heatexchanger, such that heat is transferred from the first liquid coolantto the second liquid coolant via the heat exchanger.

In other words, the method may comprise circulating a first liquidcoolant around a first cooling circulatory arrangement, and circulatinga second liquid coolant around a second cooling circulatory arrangement.Each of the first and second cooling circulatory arrangements areconfigured to cool at least respective first and second electronicdevices. Moreover, the second cooling circulatory arrangement isarranged such that the second liquid coolant receives heat transferredfrom the first liquid coolant at a heat exchanger. Advantageously, thishybrid cooling system provides the benefits of a high performancecooling system in relation to the hottest components (the second coolingcirculatory arrangement, targeting the first electronic device), butthen uses a further cooling system for cooling the other components inthe electronic module. In particular, it may not be practical to mount atargeted cooling system (such as provided by the second coolingcirculatory arrangement) to every component within the electronicmodule, and so the first cooling circulatory arrangement may provide anadditional cooling system for the remaining components and to lower thetemperature of the general environment within the electronic module.Moreover, the return flow of the high performance second coolingcirculatory arrangement can itself also be utilised to remove heat fromthe first cooling system.

It will be understood that the features discussed above with respect tothe system, can also be considered to be disclosed in use of the methodfor cooling the plurality of electronic devices housed in the housing ofthe electronic module.

In particular, preferably, the second cooling circulatory arrangementfurther comprises a cooling system, wherein the circulating the secondliquid coolant around the second cooling circulatory arrangementcomprises circulating the second liquid coolant between the secondelectronic device of the plurality of electronic devices, the heatexchanger and the cooling system, where heat is removed from the secondliquid coolant by the cooling system. The cooling system may be externalto the electronic module, and configured to transfer heat out of thesecond liquid coolant. For instance, the cooling system may comprise aheat exchanger to transfer heat to a further (third) coolant liquid ormedium.

Alternatively, the second cooling circulatory arrangement furthercomprises a second liquid coolant supply, wherein the circulating thesecond liquid coolant around the second cooling circulatory arrangementcomprises receiving the second liquid coolant from the second liquidcoolant supply, circulating the second liquid coolant between the secondelectronic device of the plurality of electronic devices and the heatexchanger, to then be returned to the second liquid coolant supply. Forinstance, the second liquid coolant may be water, and the second coolingcirculatory arrangement may be connected to a facility water supply.Once the water is circulated through the second cooling circulatoryarrangement, it may be allowed to pass into a drainage system (and sowould not be recirculated through the second cooling circulatoryarrangement).

The heat exchanger may comprise at least a first and a second chamberseparated by a thermal interface, wherein the heat exchanger isconfigured for flow of the first liquid coolant through at least thefirst chamber, and flow of the second liquid coolant through at leastthe second chamber, such that heat is transferred from the first liquidcoolant to the second liquid coolant through the thermal interface. Theheat exchanger is a specific element configured for efficient transferof heat between the first and second liquid coolant. The heat exchangermay be of any suitable design to allow the flow of the first and secondliquid coolant therethrough and to exchange heat therebetween.Optionally the heat exchanger is a plate heat exchanger.

Preferably, the heat exchanger is arranged within, or contained within,the housing of the electronic module. Beneficially, this avoids thefirst liquid coolant from passing out of the electronic module. Thisboth reduces the complexity of connections at the housing of theelectronic module, and also reduces the risk for leaks or loss of thefirst liquid coolant.

Preferably, the first cooling circulatory arrangement and the secondcooling circulatory arrangement are thermally coupled via the heatexchanger on the return flow of the second cooling circulatoryarrangement. In other words, the first cooling circulatory arrangementand the second cooling circulatory arrangement are thermally coupled viathe heat exchanger so that the second liquid coolant passes through theheat exchanger after receiving heat from the second electronic device(when considering the coolest second liquid coolant to be found at thestart of the second cooling circulatory arrangement). The first coolingcirculatory arrangement may be at least partially insulated from thesecond cooling circulatory arrangement, prior to the heat exchanger.This provides the greatest possible temperature gradient between thesecond liquid coolant and the second electronic device.

Preferably, the housing of the electronic module contains the firstliquid coolant, and wherein the first electronic device is at leastpartially immersed in the first liquid coolant. In other words, thehousing of the electronic device contains a reservoir of the firstliquid coolant, in which at least the first electronic device is atleast partially immersed. Heat is therefore transferred directly to thefirst liquid coolant from a surface of the first electronic device thatis at least partially immersed in the first liquid coolant.

Preferably the first cooling circulatory arrangement may comprise aweir. The weir may comprise:

a base and a retaining wall extending from the base, the base andretaining wall defining a volume for holding some of the first liquidcoolant;

an inlet, through which the first liquid coolant flows into the volume;

wherein flowing sufficient first liquid coolant through the inlet intothe volume causes the first liquid coolant to overflow the retainingwall and collect with first liquid coolant contained in the housing ofthe electronic module and exterior the weir.

Advantageously, the weir acts to promote flow of the first liquidcoolant within the electronic module. The weir may further be arrangedto direct the flow of the first liquid coolant circulating through thefirst cooling circulatory arrangement to specific electronic components.

In a particularly preferred example, the base of the weir may bethermally coupled to the first electronic device. In this way, the weiracts as an effective heat sink for the first electronic device. The weiralso maintains a flow of first liquid coolant which may cool heatgenerating components arranged around the first electronic device towhich the weir is coupled.

The inlet may further comprise a nozzle arrangement for directing thefirst liquid coolant flowing into the volume. The nozzle arrangementcomprises one or more nozzle.

The weir may further comprise projections extending from the base and/orretaining wall within the volume of the weir.

The method may further comprise providing a pump within the firstcooling circulatory arrangement, the pump configured to circulate thefirst liquid coolant around the first cooling circulatory arrangement.The first cooling circulatory arrangement may further comprise a pumpinlet, for receiving first liquid coolant contained in the housing ofthe electronic module and exterior the weir.

The method may further comprise providing a plurality of pipes withinthe first cooling circulatory arrangement, arranged to transport thefirst liquid coolant from the pump to the heat exchanger and the fromthe heat exchanger to the inlet of the weir, respectively.

Preferably, the method comprises providing a cooling module within thesecond cooling circulatory arrangement, configured to thermally couplethe second electronic device to the second liquid coolant. The coolingmodule may be mounted or coupled to a surface of the second electronicdevice. The cooling module may provide a mechanism for indirect transferof heat from the second electronic device to the second liquid coolant,via the cooling module (in other words, the second liquid coolant doesnot make direct contact with surfaces of the second electronic device,but instead heat is transferred from the second electronic devicethrough a portion of the cooling module, to be received at the secondliquid coolant). The cooling module may provide a higher cooling power,and assist in more efficient cooling of the second electronic devicethan could otherwise be provided by the first cooling circulatoryarrangement.

In a preferred example, providing the cooling module within the secondcooling circulatory arrangement comprises providing a cold plate, thecold plate comprising:

a cold plate housing, a surface of the cold plate housing being arrangedto provide a thermal interface for cooling the second electronic devicewhich is thermally coupled thereto; and

at least one channel within the cold plate housing and proximate to thesurface of the cold plate housing, the at least one channel arranged forthe second liquid coolant to flow therethough such that heat receivedfrom the second electronic device through the surface of the cold platehousing is transferred to the second liquid coolant.

More than one cold plate can be provided, which may be arranged inparallel or in series within the second cooling circulatory arrangement.The surface of the cold plate housing may be directly coupled to asurface of the second electronic device, or may be coupled via aninterface to promote effective heat transfer.

The method may further comprise providing, within the second coolingcirculatory arrangement, a plurality of conduits or pipes arranged totransport the second liquid coolant between the cold plate, the heatexchanger and the cooling system.

In a still further aspect, there is described a system for cooling anelectronic module (or server module), configured for installation into arack (or server rack). The system comprises at least a first and secondcold plate (or cold plate module, or cold plate assembly) mounted withinthe electronic module, through which liquid coolant (which may be water,for instance, or a dielectric fluid) circulates in a cooling loop. Thefirst and second cold plate are each arranged on separate, parallelbranches of the cooling loop. During circulation of the liquid coolantthrough the cold plate, heat is transferred to the liquid coolantpassing through each of the first and second cold plates from one ormore electronic devices thermally coupled to the cold plates. A coolingsystem is connected within the cooling loop, in order to remove heat ortransfer heat away from the liquid circulating in the cooling loop. Thecooling liquid may be arranged either outside or inside the chassis orhousing of the electronic module.

In a further aspect there is a system for cooling an electronic moduleconfigured for installation into a rack, comprising a first and a secondcold plate, mounted within a module housing of the electronic module.Each cold plate comprises a housing, wherein a surface of the housing isarranged to provide a thermal interface for cooling an electronic devicethermally coupled thereto. Each cold plate further comprises at leastone channel within the housing and proximate to the surface, arrangedfor a liquid coolant to flow therethough such that heat received by thethermal interface is transferred to the liquid coolant. The first andsecond cold plates of the system are coupled in parallel in a coolingloop, the cooling loop arranged to circulate the liquid coolant.

Preferably, the system further comprises a cooling system, the coolingsystem configured to remove heat from the liquid coolant, and aplurality of conduits, coupled to the first and second cold plate andthe cooling system for transferring the liquid coolant circulating inthe cooling loop between the first and second cold plate and the coolingsystem. The conduits may be pipes or tubing, which preferably areflexible in order to allow arrangement of the conduits around componentswithin the electronic module. The pipes may be of a small bore size ordiameter (for instance between 2 to 20 mm, and more preferably 3 to 10mm), to assist with arrangement within an existing configuration ofcomponents within the electronic module. This is also beneficial, toallow feedthrough of the pipes or tubes through existing holes, openingsor apertures in the module housing or chassis, The cooling system maycomprise at least a heat exchanger, for transfer of heat from the liquidcoolant to another cooling medium (such as a second liquid coolant, orair). The cooling system may be arranged within the housing or chassisof the electronic module, or may be arranged outside of the module.

Preferably, the plurality of conduits comprises at least a first inputconduit coupled to the first cold plate, a second input conduit coupledto the second cold plate, a supply conduit, to which the first and thesecond input conduit are coupled in parallel, a first output conduitcoupled to the first cold plate, a second output conduit coupled to thesecond cold plate, and a receiving conduit, to which the first and thesecond output conduits are coupled in parallel. In other words, theplurality of conduits comprises at least a conduits forming the firstand second parallel branches of the cooling loop (the first input andoutput conduit, and the second input and output conduits, respectively),and two further conduits to supply or receive liquid coolant to theparallel branches of the cooling loop.

The coupling between the first and the second input conduit and thesupply conduit and the coupling between the first and the second outputconduit and the receiving conduit may both be arranged within the modulehousing. In this case, the supply conduit and the receiving conduit arearranged to pass through an opening in the wall of the module housing.Alternatively, the coupling between the first and the second inputconduit and the supply conduit, and the coupling between the first andthe second output conduit and the receiving conduit may be arrangedoutside of the module housing. In this case the first and the secondinput conduit and the first and the second output conduit are arrangedto pass through an opening in the wall of the module housing.

Optionally, the system further comprises a manifold for coupling thefirst and the second input conduit to the supply conduit and/or forcoupling the first and the second output conduit to the receivingconduit. Optionally, the system comprises a manifold, for coupling thesupply conduit to a first further conduit of the plurality of conduits,and/or for coupling the receiving conduit to a second further conduit ofthe plurality of conduits. The manifold may be a unit for joining orcoupling the conduits. Beneficially, the manifold may provide a couplingthat this more robust. The manifold may be a single unit for couplingthe input conduits and the supply conduit as well as the output conduitsand receiving conduit (with appropriate partition). However, twomanifolds may be used, each associated with the supply and receivingside of the cooling loop.

The system may comprise at least one connector for connection of thefirst and the second input conduit and/or the first and the secondoutput conduit to the manifold. The system may comprise at least oneconnector for connection of the supply conduit and/or the receivingconduit to the manifold. Any type of suitable connector may be used forconnection of the conduits to the manifold. In one example, drip-free,manual connectors are used, which require direct manual manipulation byan installer. In a further example, blind mate connectors may be used,which are push fit connectors not requiring specific manipulation to fittogether and seal the connectors. Manually connected drip freeconnectors advantageously can connect with existing IT, as the tubes canexit the module housing wherever there is an opening or aperture. Blindmate connectors would require a bespoke bracket for the each electronicmodule, but may provide an easier fit.

In one example, the system may comprise a bracket for securing themanifold to the module housing. In a further example, the system maycomprise a bracket for securing the manifold to the rack. The mountingof the manifold to the electronic module to the rack may be chosenaccording to the space available in the rack, and the requirements ofthe system. Connection of the manifold to the electronic module mayreduce movement and strain on the connections between conduits. However,connection of the manifold to the rack may be more compact, and lesslikely to be knocked or encounter accidental damage.

In a particular advantageous example, the manifold may be mounted to therack within a duct defined in the rack for passage of cables connectedto the electronic module. As noted above, the electronic modules andracks typically conform to industry standards. Commonly, the racksinclude a rear cavity, portion or duct through which electronic and datacables can be housed or passed, before connection to an individualelectronic module. This cavity or duct can be used to house the one ormore manifold, mounted within. This provides a more robust system, andthe manifolds are protected and partially enclosed by the duct. Thewalls of the duct also provides a suitable fixing point for themanifolds (especially as the duct will typically include existingapertures of holes for feed through or fixture of cables).

Preferably, the system comprises a third cold plate, the third coldplate coupled in series with either the first or the second cold platewithin the cooling loop. In fact, the system can comprise more thanthree or any number of cold plates. In particular, the system mayinclude two or more cold plates on each parallel branch of the cooingloop. For instance, the first parallel branch of the cooling loop maycomprise a single first cold plate, and a second parallel branch of thecooling loop may comprise a second and a third cold plate, where thesecond and third cold plates are themselves arranged in series. Morethan two cold plates may be arranged in series on each parallel branchof the cooling loop.

Moreover, more than two parallel branches of the cooling loop may bearranged within the electronic module, each parallel branch having oneor more cold plates connected thereon. If more than one cold plate isarranged on a particular parallel branch of the cooling loop, then theadditional cold plates may be arranged in series on the given branch.

Advantageously, the described system can allow circulation of liquidcoolant through cold plates either individually, or in groups, inparallel or in series, or as a mixture of both, for instance.

In a further aspect, there is described a method for cooling anelectronic module configured for installation into a rack, comprisingcirculating a liquid coolant around a cooling loop, wherein a first andsecond cold plate are coupled in parallel within the cooling loop, andwherein the first and the second cold plate are housed within a modulehousing of the electronic module. Each of the first and the second coldplates comprise a housing, a surface of the housing being arranged toprovide a thermal interface for cooling an electronic device thermallycoupled thereto, and at least one channel within the housing andproximate to the surface, arranged for the liquid coolant to flowtherethough such that heat received by the thermal interface istransferred to the liquid coolant. Each of the features mentioned withinthis disclosure with reference to the system for cooling may beimplemented via the method for cooling of this present aspect.

The cooling system described within the present disclosure may be fittedwithin typical or standard electronic modules (or server modules), ofthe type configured for installation in a rack (or server rack). Inparticular, the described cooling system can advantageously beretro-fitted into existing electronic modules (or panels of servermodules within a server rack). The described cooling system can be usedto replace existing cooling systems, such as the air cooled systemtypical within common, or mass manufactured server modules. Such aircooled systems generally include an air cooled heat sink within thechassis of the server module, and this can be removed and replaced bythe cold plates described as part of the cooling system of the presentdisclosure. As such, the ‘foot print’ (or dimensions of each cold platein the plane of the server module, e.g. 50 mm×50 mm to 150 mm×150 mm)may be the same or similar to that of typical air cooled heat sinks usedin this type of server module.

Advantageously, the presently disclosed cooling system based on liquidcooled cold plates is more efficient and provides a greater coolingpower than the air-cooled systems it is intended to replace. Provisionof a more efficient cooling system for an electronic module or servermodule allows for a greater number of components to be included in aparticular electronic module (thus improving space efficiency) and/orfor the components within the electronic module to be operated at ahigher power rate of performance. In particular, use of the describedcold plates (and cooling system) can allow operation of componentswithin the electronic modules (such as CPU's) at a higher rate forlonger, as the described system is more effective at removing heat. Assuch, provision of the described cooling system removes some of thelimitations presently imposed on computing performance and use of spacein electronic modules or server modules, as a result of the challengesfor sufficient cooling seen in prior art systems.

Accordingly, it is particularly beneficial to retrofit the coolingsystem described in the present disclosure within to existing electronicmodules which previous utilised an alternative cooling system. As such,in a still further aspect there is provided a method of installation ofa liquid cooled system for an electronic module configured forinstallation into a rack, comprising removing an air cooled heat sinkcontained within a module housing of the electronic module, mounting acold plate within the module housing, in the former position of the aircooled heat sink, coupling an input conduit to the inlet cooling port ofthe cold plate, coupling an output conduit to the outlet cooling port ofthe cold plate, and connecting the input and the output conduits withina liquid coolant fluid loop, such that liquid coolant circulated aroundthe liquid coolant fluid loop is passed through at least the inputconduit, through the cold plate, and through the output conduit. Eachcold plate may comprise a housing, a surface of the housing beingarranged to provide a thermal interface for cooling an electronic devicethermally coupled thereto at least one channel within the housing andproximate to the surface, arranged for a liquid coolant to flowtherethough such that heat received by the thermal interface istransferred to the liquid coolant, an inlet coolant port extendingoutside the housing, for transferring liquid coolant to the at least onechannel and an outlet coolant port extending outside the housing, fortransferring liquid coolant from the at least one channel. In will beunderstood that more than one cold plate could be fitted according tothis method. In particular, two or more air cooled heat sinks could beremoved from an electronic module and replaced by respective two or morecold plates. The two or more cold plates could be fitted and arranged inparallel within the cooling loop (as discussed in above aspects), oralternatively could be arranged in series in the cooling loop.

Preferably, the method further comprises arranging the input and outputconduits to pass through an opening in a wall of the module housing.

Preferably, connecting the input and the output conduits within a liquidcoolant fluid loop may comprise coupling the input conduit to a firstexit port of a manifold, coupling the output conduit to a first entranceport of the manifold, coupling a supply conduit to a second entranceport of the manifold, for transferring liquid coolant to the manifoldfrom a heat exchanger, and coupling a receiving conduit to a second exitport of the manifold, for transferring liquid coolant from the manifoldto the heat exchanger.

Optionally, the method further comprises securing the manifold to therack. In one example, securing the manifold to the rack comprisessecuring the manifold within a duct defined in the rack for passage ofcables connected to the electronic module.

The method may further comprise connecting a heat exchanger between thesupply conduit and the receiving conduit, the heat exchanger configuredto transfer heat out of the liquid coolant. The method may also compriseconnecting a pump between the supply conduit and the receiving conduit,the pump configured to circulate the liquid coolant around the liquidcoolant fluid loop.

Alternatively, for instance when a manifold is not present within thecooling loop, the heat exchanger may be connected between the inputconduit and the output conduit. Similarly, the pump may be connectedbetween the input conduit and the output conduit. This example may beused where the full cooling loop resides within the module housing ofthe electronic module. Therefore, the pump, the cooling system (or heatexchanger) and the cold plates are all within the module housing. Thecooling system can be used to transfer heat out of the liquid coolant inthe cooling loop to another medium (such as a second cooling looparranged to pass through the housing.

In a yet still further aspect, there is disclosed a kit for use in themethod of installation described above. The kit may comprise the atleast one cold plate, a first pipe, for use as the input conduit, and asecond pipe, for use as the second conduit. The kit may comprise any ofthe component described above in relation to other aspects, forinstance. Options, characteristics and benefits noted above with respectto each component apply to this aspect.

Preferably, the inlet and the outlet coolant ports of the cold plateeach comprise an independently rotating fluid connector, therebyallowing adjustment in the direction of the first or the second pipecoupled to the respective coolant port.

Preferably, the cold plate further comprises pins and/or fins arrangedwithin the at least one channel. Optionally, the pins and/or fins arearranged to extend from a bottom surface of the at least one channelthat is proximate the surface of the housing arranged to provide thethermal interface to a top surface of the at least one channel that isdistal the surface of the housing arranged to provide the thermalinterface.

Preferably, the kit further comprises a manifold for connection of theinput and the output conduits within the liquid coolant fluid loop, themanifold comprising a first entrance port, for coupling to the outputconduit, a first exit port, for coupling to the input conduit a secondentrance port, for receiving liquid coolant at the manifold through asupply conduit, and a second exit port, for transferring liquid coolantfrom the manifold though a receiving conduit. Optionally, two separatemanifolds can be provided, for instance for coupling of the inputconduit to the supply conduit, and for coupling of the output conduit tothe receiving conduit, respectively, the two manifold having appropriateports.

Optionally, the manifold is configured to be secured to the rack, andmore preferably secured to the rack within a duct defined in the rackfor passage of cables for connection to the electronic module.

The kit may further comprise a heat exchanger and or a pump. The heatexchanger and/or pump may be for connection between the supply conduitand the receiving conduit, or for connection between the supply conduitand the receiving conduit, depending on the arrangement of the coolingloop.

In a still further example, there is a system for cooling an electronicmodule configured for installation into a rack, comprising: a firstcooling loop, through which a first liquid coolant is circulated, thefirst coolant loop comprising at least one cold plate; a second coolingloop, through which a second liquid coolant is circulated; a coolingsystem for transfer of heat from the first liquid coolant to the secondliquid coolant; wherein the first coolant loop and the cooling systemare contained within a housing of the electronic module.

The first and second cooling loop may be considered a first and secondcooling circulatory arrangement, respectively. Beneficially, thedescribed configuration allows the first liquid coolant, whichcirculates through the at least one cold plate, to be kept entirelywithin the electronic module (i.e. enclosed within the housing orchassis of the electronic module). As such, when the electronic moduleis installed or uninstalled from the rack, no connections ordisconnections are required to the first coolant loop—the first coolantloop is a ‘closed loop’ throughout the installation process.Accordingly, the risk of leakage or loss of expensive and sometimestoxic coolant fluid can be reduced or avoided. Moreover, a differenttype of coolant fluid (for example, more expensive, dielectric liquid)can be used as the first liquid coolant, with a second, more abundantlyavailable fluid (for instance, water) as the second liquid coolant. Useof water as a second liquid coolant can be helpful to allow a large flowrate and provide increased cooling power, but advantageously, thedescribed configuration can maintain the second liquid coolant at adistance from any electrical components or electronic device housedwithin the chassis.

Preferably, each cold plate comprises: a housing, a surface of thehousing being arranged to provide a thermal interface for cooling anelectronic device thermally coupled thereto; and at least one channelwithin the housing and proximate to the surface, arranged for the firstliquid coolant to flow therethough such that heat received by thethermal interface is transferred to the first liquid coolant.

Optionally, the first coolant loop comprises a first and a second coldplate, mounted within the housing of the electronic module, the firstand second cold plates coupled in parallel in the first cooling loop.

Preferably, the first cooling loop further comprises a pump, forcirculation of the first liquid coolant around the first cooling loop,the pump housed within the housing or chassis of the electronic module.

Preferably, the cooling system is a heat exchanger. Optionally, thecooling system is a first cooling system, and the second cooling loopfurther comprises a second cooling system, for transferring heat out ofthe second liquid coolant.

Preferably, the second cooling loop further comprises a pump, forcirculation of the second liquid coolant around the second cooling loop.

The following numbered clauses show illustrative examples only:

1. A system for cooling an electronic module configured for installationinto a rack, comprising:

a first and a second cold plate, mounted within a module housing of theelectronic module, each cold plate comprising:

-   -   a housing, a surface of the housing being arranged to provide a        thermal interface for cooling an electronic device thermally        coupled thereto; and    -   at least one channel within the housing and proximate to the        surface, arranged for a liquid coolant to flow therethough such        that heat received by the thermal interface is transferred to        the liquid coolant;

wherein the first and second cold plates are coupled in parallel in acooling loop, the cooling loop arranged to circulate the liquid coolant.

2. The system of clause 1, further comprising:

a cooling system, the cooling system configured to remove heat from theliquid coolant; and

a plurality of conduits, coupled to the first and second cold plate andthe cooling system for transferring the liquid coolant circulating inthe cooling loop between the first and second cold plate and the coolingsystem.

3. The system of clause 2, wherein the cooling system comprises a heatexchanger for transfer of heat from the liquid coolant to a furthercooling medium.4. The system of clause 2 or clause 3, wherein the plurality of conduitscomprises at least:

a first input conduit coupled to the first cold plate,

a second input conduit coupled to the second cold plate,

a supply conduit, to which the first and the second input conduit arecoupled in parallel;

a first output conduit coupled to the first cold plate;

a second output conduit coupled to the second cold plate; and

a receiving conduit, to which the first and the second output conduitsare coupled in parallel.

5. The system of clause 4, wherein the coupling between the first andthe second input conduit and the supply conduit is arranged within themodule housing;

the coupling between the first and the second output conduit and thereceiving conduit is arranged within the module housing; and

the supply conduit and the receiving conduit are arranged to passthrough an opening in the wall of the module housing.

6. The system of clause 4, wherein the coupling between the first andthe second input conduit and the supply conduit is arranged outside ofthe module housing;

the coupling between the first and the second output conduit and thereceiving conduit is arranged outside of the module housing; and

the first and the second input conduit and the first and the secondoutput conduits are arranged to pass through an opening in the wall ofthe module housing.

7. The system of clause 4 or clause 6, further comprising a manifold,for coupling the first and the second input conduit to the supplyconduit and/or for coupling the first and the second output conduit tothe receiving conduit.8. The system of clause 7, further comprising at least one connector forconnection of the first and the second input conduit and/or the firstand the second output conduit to the manifold.9. The system of any one of clauses 4 to 6, further comprising amanifold, for coupling the supply conduit to a first further conduit ofthe plurality of conduits, and/or for coupling the receiving conduit toa second further conduit of the plurality of conduits.10. The system of clause 9, further comprising at least one connectorfor connection of the supply conduit and/or the receiving conduit to themanifold.11. The system of any one of clauses 7 to 10, further comprising abracket for securing the manifold to the module housing.12. The system of any one of clauses 7 to 11, further comprising abracket for securing the manifold to the rack.13. The system of clause 12, wherein the manifold is configured to besecured to the rack within a duct defined in the rack for passage ofcables connected to the electronic module.14. The system any preceding clause, further comprising a third coldplate, the third cold plate coupled in series with either the first orthe second cold plate within the cooling loop.15. A method for cooling an electronic module configured forinstallation into a rack, comprising:

circulating a liquid coolant around a cooling loop, wherein a first andsecond cold plate are coupled in parallel within the cooling loop, andwherein the first and the second cold plate are housed within a modulehousing of the electronic module, and further wherein each of the firstand the second cold plate comprises:

-   -   a housing, a surface of the housing being arranged to provide a        thermal interface for cooling an electronic device thermally        coupled thereto; and    -   at least one channel within the housing and proximate to the        surface, arranged for the liquid coolant to flow therethough        such that heat received by the thermal interface is transferred        to the liquid coolant.        16. A method of installation of a liquid cooled system for an        electronic module configured for installation into a rack,        comprising;

removing an air cooled heat sink contained within a module housing ofthe electronic module;

mounting a cold plate within the module housing, in the former positionof the air cooled heat sink, the cold plate comprising:

-   -   a housing, a surface of the housing being arranged to provide a        thermal interface for cooling an electronic device thermally        coupled thereto;    -   at least one channel within the housing and proximate to the        surface, arranged for a liquid coolant to flow therethough such        that heat received by the thermal interface is transferred to        the liquid coolant;    -   an inlet coolant port extending outside the housing, for        transferring liquid coolant to the at least one channel; and    -   an outlet coolant port extending outside the housing, for        transferring liquid coolant from the at least one channel;

coupling an input conduit to the inlet cooling port of the cold plate;

coupling an output conduit to the outlet cooling port of the cold plate;and

connecting the input and the output conduits within a liquid coolantfluid loop, such that liquid coolant circulated around the liquidcoolant fluid loop is passed through at least the input conduit, throughthe cold plate, and through the output conduit.

17. The method of clause 16, further comprising:

arranging the input and output conduits to pass through an opening in awall of the module housing.

18. The method of clause 16 or clause 17, wherein connecting the inputand the output conduits within a liquid coolant fluid loop comprises:

coupling the input conduit to a first exit port of a manifold;

coupling the output conduit to a first entrance port of the manifold;

coupling a supply conduit to a second entrance port of the manifold, fortransferring liquid coolant to the manifold from a heat exchanger; and

coupling a receiving conduit to a second exit port of the manifold, fortransferring liquid coolant from the manifold to the heat exchanger.

19. The method of clause 18, further comprising securing the manifold tothe rack.20. The method of clause 19, wherein securing the manifold to the rackcomprises securing the manifold within a duct defined in the rack forpassage of cables connected to the electronic module.21. The method of any one of clauses 18 to 20, further comprising:

connecting the heat exchanger between the supply conduit and thereceiving conduit, the heat exchanger configured to transfer heat out ofthe liquid coolant.

22. The method of any one of clauses 18 to 21, further comprising:

connecting a pump between the supply conduit and the receiving conduit,the pump configured to circulate the liquid coolant around the liquidcoolant fluid loop.

23. The method of clause 16 or 17, further comprising:

connecting the heat exchanger between the input conduit and the outputconduit, the heat exchanger configured to transfer heat out of theliquid coolant.

24. The method of clause 16, clause 17 or clause 23, further comprising:

connecting a pump between the input conduit and the output conduit, thepump configured to circulate the liquid coolant around the liquidcoolant fluid loop.

25. A kit for use in the method of installation of clauses 16 to 24,comprising:

the cold plate;

a first pipe, for use as the input conduit; and

a second pipe, for use as the second conduit.

26. The kit of clause 25, wherein the inlet and the outlet coolant portsof the cold plate each comprise an independently rotating fluidconnector, thereby allowing adjustment in the direction of the first orthe second pipe coupled to the respective coolant port.27. The kit of clause 25 or clause 26, wherein the cold plate furthercomprises pins and/or fins arranged within the at least one channel.28. The kit of clause 27, wherein the pins and/or fins are arranged toextend from a bottom surface of the at least one channel that isproximate the surface of the housing arranged to provide the thermalinterface to a top surface of the at least one channel that is distalthe surface of the housing arranged to provide the thermal interface.29. The kit of any one of clauses 25 to 28, further comprising amanifold for connection of the input and the output conduits within theliquid coolant fluid loop, the manifold comprising:

a first entrance port, for coupling to the output conduit;

a first exit port, for coupling to the input conduit;

a second entrance port, for receiving liquid coolant at the manifoldthrough a supply conduit; and

a second exit port, for transferring liquid coolant from the manifoldthough a receiving conduit.

30. The kit of clause 29, wherein the manifold is configured to besecured to the rack within a duct defined in the rack for passage ofcables for connection to the electronic module.31. The kit of clause 29 or clause 30, further comprising a heatexchanger, for connection between the supply conduit and the receivingconduit.32. The kit of any one of clauses 29 to 31, further comprising a pump,for connection between the supply conduit and the receiving conduit, thepump configured for circulating the liquid coolant around the liquidcoolant fluid loop.33. The kit of any one of clause 25 to 29, further comprising a heatexchanger, for connection between the input conduit and the outputconduit.34. The kit of any one of clauses 25 to 29 or clause 33, furthercomprising a pump, for connection between the supply conduit and thereceiving conduit, the pump configured for circulating the liquidcoolant around the liquid coolant fluid loop.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may be put into practice in a number of ways andpreferred embodiments will now be described by way of example only andwith reference to the accompanying drawings, in which:

FIG. 1 shows a plan view of an electronic module housing a plurality ofelectronic devices cooled by the described system;

FIG. 2 depicts a perspective view of the electronic module of FIG. 1;

FIG. 3 depicts a second, different perspective view of the electronicmodule of FIGS. 1 and 2;

FIG. 4A depicts a schematic view of a first example of the secondcooling circulatory arrangement;

FIG. 4B depicts a schematic view of a second example of the secondcooling circulatory arrangement;

FIG. 5 depicts a schematic view of a plurality of electronic modulesmounted in a rack;

FIG. 6 illustrates a perspective view of a weir within the first coolingcirculatory arrangement;

FIG. 7 illustrates an exploded view of the weir within the first coolingcirculatory arrangement;

FIG. 8 illustrates a cross-sectional view of the weir within the firstcooling circulatory arrangement;

FIG. 9 illustrates a further view of the weir within the first coolingcirculatory arrangement;

FIG. 10 illustrates a further example of a weir within the first coolingcirculatory arrangement;

FIG. 11 illustrates a still further example of a weir within the firstcooling circulatory arrangement;

FIG. 12A shows a schematic, perspective view of an example cold plate;

FIG. 12B shows a plan view of the internal structure of the example coldplate of FIG. 12A;

FIG. 12C shows a cross-sectional view of the example cold plate of FIG.12A;

FIG. 13A depicts a schematic view of a further example of the system;

FIG. 13B depicts a schematic view of a still further example of thesystem;

FIG. 14 depicts a plan view of an example electronic module or bladeserver with an example cold plate mounted within;

FIG. 15 depicts a perspective view of a second example of an electronicmodule or blade server with two example cold plates mounted within;

FIG. 16 illustrates a plan view of the second example of the electronicmodule or blade server of FIG. 15;

FIG. 17 illustrates a plan view of a third example of an electronicmodule or blade server with example cold plates mounted within;

FIG. 18 illustrates a plan view of a fourth example of an electronicmodule or blade server with example cold plates mounted within;

FIG. 19 illustrates a plan view of a fifth example of an electronicmodule or blade server with example cold plates mounted within;

FIG. 20 illustrates a plan view of a sixth example of an electronicmodule or blade server with example cold plates mounted within;

FIG. 21 illustrates a plan view of a seventh example of an electronicmodule or blade server with example cold plates mounted within,including a manifold;

FIG. 22 illustrates a plan view of an eighth example of an electronicmodule or blade server with example cold plates mounted within,including a manifold;

FIG. 23 a schematic view of an example of a single rack server coolingsystem in accordance with the disclosure;

FIG. 24 a schematic view of a further example of a single rack servercooling system in accordance with the disclosure; and

FIG. 25 shows a schematic view of a multi-server cooling system inaccordance with the disclosure.

In the drawings, like parts are denoted by like reference numerals. Thedrawings are not drawn to scale.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

There is described an electronic module 100 having a hybrid coolingsystem, including two cooperating circulatory loops of liquid coolant.Referring first to FIG. 1, there is depicted a preferred embodiment ofan electronic module 100, which may be a module or server blade, havingappropriate dimensions and outer connectors to fit within a commonplaceserver rack (not shown). The same electronic module 100 is shown in FIG.2 and FIG. 3, which each depict different perspective views of themodule.

The electronic module has an outer housing or enclosure 110, has a base,walls and lid, and may be sealable. A plurality of electronic devices(or heat generating components) are mounted within the housing. In somecases, the components may be mounted on printed circuit boards (PCB) 120which may be connected to the base, lid or a wall of the housing. Thedescribed system looks to remove heat generated by the electronicdevices from within the electronic module.

A first cooling circulatory arrangement (or first cooling loop) is usedto cool certain electronic devices of the plurality of electronicdevices mounted within the electronic module. A second coolingcirculatory arrangement (or second cooling loop) is used to cool othersof the heat generating components. For instance, the second coolingcirculatory arrangement can be provided with a greater cooling power,and so be used to cool specific components which generate greateramounts of heat than those components cooled by the first coolingcirculatory arrangement.

In the example depicted in FIGS. 1, 2 and 3, an electronic module has afirst cooling circulatory arrangement (or first cooling loop) thatprovides immersive cooling. The first cooling circulatory arrangement iscontained entirely within the housing 110 of the electronic module. Inparticular, a first liquid coolant is contained within the sealablehousing of the electronic module, so that a number of components to becooled 115 are at least partially immersed in the first liquid coolant.The first liquid coolant contained in the volume of the housing of theelectronic module may be considered as a reservoir of first liquidcoolant.

First liquid coolant from the reservoir of first liquid coolant iscollected or received at a pump input 190. The pump input 190 may beshaped in order to improve flow of the liquid coolant towards a pump185. The pump 185 moves the first liquid coolant through the firstcooling circulatory arrangement (or first cooling loop). First liquidcoolant passed through the pump 185 is moved through a pipe 195 and intoa heat exchanger 170, where it will be cooled. In particular, heatretained in the first coolant fluid maybe transferred to the secondcoolant fluid, which is also passed through the heat exchanger, asdescribed below. As will be understood, first liquid coolant enteringthe heat exchanger is at a higher temperature to the first liquidcoolant passing out of the heat exchanger.

In the example of FIGS. 1, 2 and 3, a pipe 200 is connected to the heatexchanger to carry first cooling fluid that is output from the heatexchanger 170. At the distal end of the pipe 200 are one or more outletsor nozzles 205 a, 205 b. In the specific example of FIGS. 1, 2 and 3,the outlets or nozzles 205 a each form the inlet to a weir 202 a, 202 b.The weirs acts as heat sinks, and are part of the first coolingcirculatory arrangement. The weir is described below in further detailwith respect to FIGS. 6 to 11.

First cooling fluid passes out of outlets or nozzles 205 a, 205 b andthrough the weir 202 a, 202 b, until being collected within thereservoir of first liquid coolant contained within the volume of thehousing 110 of the electronic module 100. In this way, the cooler firstcooling fluid which has passed through the heat exchanger can bereintroduced to the bath or reservoir of the first cooling fluid withinthe housing 110, and cool any electronic components partially immersedin the reservoir. Specifically, the cooled first liquid coolant willabsorb heat from the surfaces of the electronic devices (including thefirst electronic device with which it is in contact). Eventually, thefirst liquid coolant will once again be collected at the pump input,thereby completing the first cooling circulatory arrangement (or firstcooling loop).

FIGS. 1, 2 and 3 also depict a second cooling circulatory arrangement.The second cooling circulatory arrangement incorporates one or more coldplates 125 a, 125 b, which are each mounted to one or more electronicdevices 130 a, 130 b. Ideally, the electronic devices 130 a, 130 brequire higher performance cooling. The cold plates 125 a, 125 b are amodule or chamber, through which a second coolant fluid (such as water)can be passed. Heat can be transferred to the second coolant fluidwithin the cold plate from the electronic devices 130 a, 130 b, byconduction of the heat through the mounting surface of the cold plate,coupled to the given electronic device. The cold plates 125 a, 125 b ofthe second cooling circulatory arrangement are discussed in more detailbelow, with respect to FIGS. 12A, 12B and 12C.

The second cooling circulatory arrangement of FIGS. 1, 2 and 3 has twocold plates, connected in parallel. In particular, a single inputconduit 135 is connected to an inlet 145 at the wall of the housing, inorder to receive a second coolant fluid input to the electronic module.The inlet 145 comprises a connector, which may be any suitable type ofconnector, including a quick disconnect connector. At a distal end, thesingle input conduit 135 is connected to an input manifold 150, to whichtwo further input conduits 140 a, 140 b are connected. The further inputconduits 140 a, 140 b are each connected to a respective cold plate 125a, 125 b. In this way, the second coolant fluid can be transported, inparallel, to each of the cold plates 125 a, 125 b within the electronicmodule. The second liquid coolant is then passed through the cold plate,as discussed below in relation to FIGS. 12A, 12B and 12C.

An output conduit 155 a, 155 b is connected to each of the respectivecold plates 125 a, 125 b. The output conduits 155 a, 155 b receive thesecond liquid coolant output from each of the cold plates 125 a, 125 b,in parallel. The output conduits 155 a, 155 b are connected to an outputmanifold 160, to which is also connected a single output conduit 165,for transporting the second liquid coolant out of the output manifold160.

The single output conduit 165 is connected to the heat exchanger 170discussed above with respect to the first cooling circulatoryarrangement. The heat exchanger 170 is arranged entirely within thehousing 110 of the electronic module. In the particular example of FIGS.1, 2 and 3, the heat exchanger 170 is a plate heat exchanger andconnected to the wall of the housing 110. However, other suitable typesof heat exchanger (as discussed further below) could be used, and couldbe arranged anywhere within the electronic module (and, less preferably,outside the electronic module).

The heat exchanger 170 may be of any suitable type that allows exchangeof heat between the first and second coolant fluid whilst maintainingseparation (not intermingling) of the two liquid coolants. For instance,the heat exchanger may have a first chamber, through which the firstliquid coolant flows, which is separated from a second chamber, throughwhich the second coolant fluid flows. The wall or walls separating thefirst and second chamber acts as a thermal interface, through which heatcan be transferred. In particular, heat can be transferred from thehotter liquid coolant (which in the present example, will be the secondliquid coolant under normal operation) to the cooler liquid coolant(which in the present example, will be the first liquid coolant undernormal operation), as a result of the temperature gradient across thethermal interface. As can be envisaged by a person skilled in the art,more than two chambers could be included within the heat exchanger, andmore than one thermal interface could be provided to separate thechambers through which the different liquid coolants flow. The heatexchanger may comprise fins or other features at the thermal interfaceto promote heat exchange.

Turning back to FIGS. 1, 2 and 3, an output pipe 175, connected to theheat exchanger 170 is arranged to receive the second liquid coolantpassed through the heat exchanger. As will be understood, the secondliquid coolant passing out of the heat exchanger 170 will be at a highertemperature than the second liquid coolant entering the heat exchanger170, as a result of the heat absorbed from the first liquid coolantwithin the heat exchanger 170. The output pipe 175 is connected to anoutlet 180 at the wall of the housing 110 of the electronic module. Theoutlet 180 comprises a connector, which may be any suitable type ofconnector, including a quick disconnect connector.

Although not shown in FIGS. 1, 2 and 3, the inlet 145 and the outlet 180may each be connected to a cooling system or second liquid coolantsupply (for instance by further pipes, connected to the inlet 145 andoutlet 180, as necessary). This is discussed in further detail below, inrelation to FIGS. 4A and 4B.

As will be understood, different devices within an electronic module mayproduce different amounts of heat than other components, and so requirea different rate of cooling. Therefore, the present invention provides acooling system which delivers effective and efficient cooling for allelectronic devices within an electronic module. In particular, thesecond cooling circulatory arrangement may provide high performancecooling of the hottest components, whilst the first cooling circulatoryarrangement may provide cooling for other components within theelectronic module. Although certain previous systems have described useof a first and second cooling loop within an electronic module (forinstance, as described in U.S. Pat. No. 7,724,524), the use of a heatexchanger to exchange heat between the first and second coolingcirculatory arrangement in the present application improves the overallefficiency of cooling. In contrast, in prior art systems, cooling may belimited by the exposure of the reservoir of a first coolant to the coldplate cooled by the second coolant.

FIGS. 4A and 4B show a schematic view of the first and second coolingcirculatory arrangement, when the electronic module 100 is connected ina rack 400. The arrangement of the first and second cooling circulatoryarrangements, within the electronic modules are the same for both theexamples of FIGS. 4A and 4B and align with the first and second coolingcirculatory arrangements of FIGS. 1, 2 and 3 (except for the provisionof only a single weir 202 in the examples of the first coolingcirculatory systems of FIGS. 4A and 4B). However, the apparatus forproviding low temperature second liquid coolant to the electronicmodules is different within the two examples of FIGS. 4A and 4B.

Considering the aspects in common first, FIGS. 4A and 4B shows the firstcooling circulatory system as described above, having a pump inlet 190,pump 185, heat exchanger 170, and weir 202 for circulation of firstliquid coolant 401 within the housing of the electronic module 100.FIGS. 4A and 4B further shows the second cooling circulatory arrangementas described above. In this arrangement, the second liquid coolant 402is received into the electronic module 100 though an inlet 145, andthrough conduits to provide the coolant to two cold plates 125 a, 125 b,arranged in parallel. After passing through the cold plates 125 a, 125b, the second liquid coolant is transported through various conduits tothe heat exchanger 170, where heat is transferred from the first liquidcoolant to the second liquid coolant. From the heat exchanger 170, thesecond liquid coolant is passed out of the electronic module via anoutlet 180. The inlet 145 and outlet 180 may each be connected to aninlet 405 or outlet 410 manifold, respectively, at the server rack 400in which the electronic module 100 is mounted.

In the example of FIG. 4A, a cooling system 450 is connected as part ofthe second cooling circulatory arrangement. In particular, second liquidcoolant exiting from the outlet 180 of the electronic module and passedto an outlet manifold 410 is input to a cooling system 450. The coolingsystem 450 comprises a heat exchanger 455 and a pump 460. Heat istransferred out of the second liquid coolant to a further cooling medium470 at the heat exchanger 455, wherein the cooled second liquid coolantis pumped back to the inlet manifold 405 and the inlet 145 of theelectronic module 100. The further cooling medium 470 may be air, orfurther liquid coolant, for instance.

In the example of FIG. 4B, the second cooling circulatory arrangementcan be provided with lower temperature second liquid coolant by afacility level provision. In particular, the second liquid coolant iswater, and fed from the mains water supply 490 at a facility level. Thehigher temperature second liquid coolant received from the electronicmodule can be passed to the mains drainage 495 at the facility level.

FIG. 5 shows an example of a number of electronic modules 100 accordingto the examples FIGS. 1, 2 and 3 mounted in a rack 400 of FIG. 4A. Here,the second cooling circulatory arrangement is connected to a rack levelcooling system 450. In particular, a number of electronic modules areconnected in parallel, with a single cooling system 450 (as describedabove in relation to FIG. 4A) used to cool the secondary liquid coolantsupplied to each of the electronic modules 100 within the rack 400.

It will be understood that the second cooling circulatory arrangement ofa plurality of electronic modules mounted in a rack 400 in the mannershown in FIG. 5, could also be supplied with lower temperature secondliquid coolant by connection to a facility level cooling system, such asa facility level cooling water system fed by mains water shown in FIG.4B.

The weir of the first cooling circulatory system will now be describedin more detail with reference to FIGS. 6 to 11. The weir providesparticular advantages for directing and increasing the flow of the firstliquid coolant in the first cooling circulatory arrangement. By use ofthe weir, the first liquid coolant can be directed to flow over orthrough specific regions of the electronic module, and any electronicdevices mounted therein. In some examples, the base of the weir may becoupled to the first electronic device (or another electronic device),and therefore act as a heat sink for the coupled device. In addition,use of the weir allows the level of first liquid coolant required in theelectronic module to be reduced, as discussed further below.

Referring first to FIG. 6, there is illustrated a first embodiment of aweir or weir heat sink for use in the first cooling circulatory system.With reference to FIG. 7, there is shown an exploded view of theembodiment of FIG. 6. The weir 600 comprises: a base made up of a mount610 and a planar substrate 615 fixed to the mount 610; a retaining wall620 attached to the planar substrate 615; projections (shown in the formof pins) 625; and fixing screws 630, which attach the substrate 615 tothe mount 610. In this way, the planar substrate 615 sits directly on ahigh temperature component, which may be the first electronic device635. As such, heat is transferred from the first electronic device 635to a volume defined by the planar substrate 615 and the retaining wall620, in which projections 625 are provided.

The weir heat sink 600 can be made from a single component, for exampleby: die cast; lost wax casting; metal injection mould (MIM); additivemanufacture; or forged. It could also be machined out of a block ofmaterial or skived. The weir heat sink 600 may be formed from anymaterial that is thermally conductive, such a metal or other thermalconductor. Some examples may include aluminium, copper or carbon.

Also shown in FIGS. 6 and 7 are a pipe 640 and an inlet to the weir at anozzle 645. The first liquid coolant is delivered to the weir heat sink600 via the nozzle 645. The nozzle 645 is arranged to direct coolantperpendicular to the plane of the substrate 615. This forces the jet orflow of the liquid coolant directly into the volume defined by thesubstrate 615 and retaining wall 620 of the heat sink 600. As aconsequence, the heat dissipation is improved. This is especially thecase in comparison with a system where coolant is directed to flow overa heat sink, in a direction parallel to the plane of the heat sinksubstrate, such as in an air cooled system.

In the examples shown in FIGS. 6 and 7, the nozzle 645 delivers thecoolant directly in the centre of the volume defined by substrate 615and retaining wall 620. In this example, the centre of that volumecorresponds with the hottest part of the area of the substrate 615,which is adjacent to (and directly on) the high temperature component635. This provides a contraflow, such that the coldest coolant isdirected to contact the hottest area of the weir heat sink. The coolantmoves out radially from the hottest part.

With reference to FIG. 8, there is shown a cross-sectional view of theweir heat sink in FIG. 6 in operation. The same features as shown inprevious drawings are identified by identical reference numerals. Anarrow indicates the flow of coolant within the pipe 640, to providefirst liquid coolant 805 within the volume defined by the substrate 615and retaining wall 620 of heat sink 600 and first liquid coolant 810outside the heat sink 1. As indicated previously, first liquid coolantemerging from nozzle 645 is directed towards the centre of the volume(corresponding with the centre of the surface area of substrate 615) andfrom there moves out radially towards the retaining wall 620. Sufficientfirst liquid coolant is pumped via nozzle 645 into the volume, such thatit overflows 810 the retaining wall 620 and collects with remainingfirst liquid coolant 815 exterior to weir heat sink 600.

The retaining wall 620 acting as a side wall enables different levels ofcoolant. The first liquid coolant 805 within the volume of the weir heatsink 600 is at a relatively high level and the coolant 815, which atleast partially immerses a plurality of other electronic devices in theelectronic module (not shown in this drawing), is at a lower level. Thisallows significantly less liquid coolant to be used than in othersimilar systems that cover all components at the same height.

A number of benefits are thereby realised. Firstly, if a dielectriccoolant is used as the first liquid coolant, less first liquid coolantis used. This has two main benefits: dielectric coolant can be expensiveand so costs can be significantly reduced, and dielectric liquidcoolants are typically very heavy, and so weight of the electronicmodule can be reduced. Moreover, by using less liquid coolant, theelectronic module 100 can be more straightforward to install and/orlift. Also, installing the electronic module 100 can require lessinfrastructure. In addition, the electronic module 100 is easier tohandle than similar devices are systems using significantly more primaryliquid coolant. The level of the first liquid coolant 815 within themajority of the container 110 is not close to the top of the container.As a result, spillages during maintenance or exchange of components areless likely. The risk of leakage is also reduced.

The retaining wall 630 creates a weir effect, and promotes flow of thefirst liquid coolant. The coolant 815 at a relatively low level coolsthe electronic devices in the electronic module 100 (the firstelectronic device, and any other electronic device). It is not necessaryfor the first electronic device, and any other electronic devices, to befully immersed in first liquid coolant. The first liquid coolantretained in the weir heat sink 600 may also provide some redundancy tothe cooling of the first cooling circulatory arrangement, in the eventof a failure of the pump 185 or other component.

Referring next to FIG. 9, there is shown a top view of the embodiment ofFIG. 7, showing a nozzle arrangement. As previously discussed, thenozzle 645 is coupled to pipe 640. The nozzle 645 is positioned to facethe centre of the surface area of the substrate 615 (not shown in thisfigure). The radial flow of coolant is shown by arrows in this drawing.

Alternative positions for the nozzle 640 are possible. Some suchpositions will now be described with reference to FIG. 10, in whichthere is shown a top view of the first variant of the nozzle arrangementof the embodiment of FIG. 6 and with reference to FIG. 11, in whichthere is shown a top view of a second variant of the nozzle arrangementof the embodiment of FIG. 6. Referring first to FIG. 10, the nozzle 645is shown off-centre. Such an arrangement may be provided if the hottestpart of the first electronic device (not shown, to which the weir heatsink 600 may be coupled) is not adjacent the centre of the substrate615. Referring to FIG. 11, two nozzles are shown. The two nozzles 645are positioned over the surface area of the substrate 615 (not shown)adjacent two of the hottest parts of the first electronic device (notshown, to which the weir heat sink 600 may be coupled).

The projections 625 (as pin and/or fins) could integrally be formed withthe rest of weir heat sink 600 or be made from separate components. Theprojections 625 could be tolerance fit, glued or brazed in place.Additionally or alternatively, the retaining wall 620 could beintegrally formed or made separately from the rest of the heat sink 600,for example by an extrusion or fabricated sheet metal part. Then, theretaining wall 620 could be tolerance fit, glued in place, brazed orwelded.

The cold plate of the second cooling circulatory arrangement will now bedescribed in more detail with reference to FIGS. 12A to 12C. The coldplate has particular advantages to provide high performance, efficientcooling of specific electronic devices to which the cold plate iscoupled. As such, the second cooling circulatory arrangement, and morespecifically the cold plates, can be coupled to the electronic devicesin the electronic module which produce the greatest amount of heat.Although it would not be practical to cool every electronic devicewithin the electronic module in this way, the use of cold plates as partof the second cooling circulatory arrangement allows focussed coolingwhich can reduce the burden of cooling the whole volume of theelectronic module by the first cooling circulatory system. As such, thefirst and the second cooling circulatory arrangement act in cooperationto provide a particularly efficient and effective cooling system for theelectronic module.

A further benefit of the cold plates within the second coolingcirculatory arrangement is the provision of a closed, sealed system,wherein the second liquid coolant does not make direct contact with anyelectronic device. This allows use of water for the second liquidcoolant (rather than dielectric liquid, for instance), which is readilyavailable at a low cost. A large throughput of water through the secondcooling circulatory arrangement is possible if the second coolingcirculatory arrangement is connected to a facility water supply anddrainage, or to a powerful pump system external to the electronicmodule, which further increases the potential cooling power of thesecond cooling circulatory arrangement.

In general terms, there is herein described a cold plate, comprising ahousing (which may be integrally formed), a surface of the housing(typically planar) being arranged to provide a thermal interface (whichmay be termed a conduction surface) for cooling an electronic devicethermally coupled thereto. The cold plate further comprises at least onechannel within the housing and proximate to the surface. The channel orchannels may be formed of an internal chamber (or chambers), volume orother space for containing liquid coolant (such as water, a water-basedcoolant, a coolant that essentially comprises water or a highspecific-heat capacity liquid alternative such as a mineral oil ordielectric fluid). The channel or channels are arranged for the liquidcoolant to flow therethough, such that heat received by the thermalinterface is transferred to the liquid coolant. Optionally, a pluralityof parallel channels may be provided, each extending from the coolantport. As will be discussed further below, pins and/or fins arepreferably arranged within the at least one channel.

The cold plate also comprises a coolant port extending outside thehousing, for transferring liquid coolant to and/or from the at least onechannel. The coolant port may be a connector, coupling, joint or othersimilar structure. An inlet and an outlet coolant port may be provided.At least one conduit, such as a pipe, hose or tube (preferablyflexible), may be coupled to the coolant port for transferring liquidcoolant to and/or from the coolant port. Advantageously, the cold plateis configured such that the liquid coolant remains substantially in aliquid state (that is, single phase liquid cooling) throughout thecooling system.

Referring first to FIG. 12A, there is schematically shown an embodimentof a cold plate (or cold plate assembly) 1200 for use as the coolingmodule of the second cooling circulatory arrangement. In particular, thecold plate is advantageous for use in an electronic module or serverblade (or similar module). The cold plate assembly comprises: a coldplate housing 1210 (preferably made integrally); connectors 1220 a, 1220b forming an inlet/outlet port of the cold plate; and inlet/outletconduits 1225 (which here are pipes or tubes). Also shown are fixturepoints 1230 for the cold plate 1200. These fixture points maybeneficially replicate the ones found on air cooled heatsinks in atypical server chassis, allowing the cold plates to be retrofitted intoa server blade.

In a preferred example, multiple coolant ports 1220 a, 1220 b are used,as shown in FIG. 12A. Then, a first coolant port 1220 a is provided fortransferring liquid coolant into the cold plate 1200, and a secondcoolant port 1220 a is provided for transferring liquid coolant from thecold plate 1200. In the example of FIG. 12A, the coolant ports 1220 a,1220 b are coupled to independently rotating fluid connectors (or swiveljoints or swivel elbow connectors, the terms being used synonymouslyherein) thereby allowing adjustment in the direction of the inlet/outletconduit 1225 a, 1225 b coupled to the coolant port 1220 a, 1220 b. Assuch, this type of connector are especially useful in configuring orinstalling the cold plate assembly for operation. The swivel joint mayincrease flexibility of placement of the cold plate. This may morereadily allow retro-fitting of the cold plate to an existing electronicmodule, such as a server or other computer system, without the need tomake any other changes to the unit or system. The cold plate may beconfigured to fit in place of an air-cooled heat sink, for example.

Preferably, the surface arranged to provide a thermal interface to anelectronic device (such as the second electronic device) to which thecold plate is coupled is a bottom surface of the cold plate housing (forinstance the underside, not shown, of the cold plate housing 1210 inFIG. 12A). Then, the coolant ports 1220 a, 1220 b are advantageouslyprovided on a top surface of the housing 1210, opposite the bottomsurface. In preferred embodiments, the coolant port extends in adirection perpendicular to the top surface of the housing. The swiveljoint may then extend the coolant port in a different direction,typically more parallel to the top surface of the housing. Beneficially,the swivel joint allows the direction of the pipe to be adjusted aroundan axis perpendicular to the top surface of the housing. In particular,the swivel joint may allow the direction of the pipe to be adjustedthrough at least 90 degrees, 180 degrees, 270 degrees and preferably upto (and including) 360 degrees, especially around an axis perpendicularto the top surface of the housing. Thus, the swivel connector may allowfull rotation freedom for the coolant port.

In principle, a single coolant port could provide both an inlet forliquid coolant to the channel and an outlet for liquid coolant from thechannel. In the preferred embodiment, multiple coolant ports are used,as shown in FIG. 12A. Then, the coolant port is a first coolant port fortransferring liquid coolant to the at least one channel. The cold platemay comprise a second coolant port for transferring liquid coolant fromthe at least one channel.

The housing of the cold plate, and the arrangement of the ports on thecold plate may take any shape beneficial to promote flow of coolantthrough the cold plate and to portions of a second electronic device towhich the cold plate is coupled. In some examples, the housing iselongated and the first and second coolant ports are located at oppositeends of the housing along the direction of elongation, which may promoteflow of liquid coolant across the thermal interface surface and/or aidflexible placement of the cold plate. Additionally or alternatively, thesecond coolant port may (like the first coolant port) comprise a swiveljoint thereby allowing adjustment in the direction of a pipe coupled tothe second coolant port. Providing two coolant ports, each with swiveljoints, may allow improved ways of coupling the cold plate within acooling system, including the potential to couple cold plates together.

With reference to FIG. 12B, there is depicted a top internal (plan) viewof an example cold plate in accordance with the example of FIG. 12A. Inthis drawing, the lid and nozzles are removed for clarity.

Shown in FIG. 12B are: coolant inlet port 1225 a; coolant outlet orexhaust port 1225 b; coolant flow channel 1235; and pins 1240. It can beseen that the flow channel is formed between a coolant inlet port 1225 aand a coolant outlet or exhaust port 1225 b, through which, when thecold plate is in use, the second liquid coolant can flow. Thisconfiguration, and especially the configuration of the pins 1240,distributes coolant flow in all directions within the cold plate,allowing coolant to spread evenly across the cold plate.

Referring to FIG. 12C, there is shown a side (cross-sectional) view ofthe embodiment of FIG. 12B, with connectors coupled to ports 1225 a,1225 b, cold plate base 1245. Cold plate lid 1250 is also shown. Thebase plate 1245 and lid 1250 may together form the cold plate housing1210 shown in FIG. 12A. As shown in FIG. 12C, the pins 1240 areconnected to the base 1245 and lid 1250. This may ensure that no flow ofliquid coolant passing through the channel 106 can short-cut or bypassthe pins 109. In this way, the pins 109 can direct the coolant flowwithin the cold plate.

The base plate 1245 may provide the thermal interface, to which thesecond electronic device can be coupled. In particular, the cold plate1200 can be mounted to an electronic device (for instance, a second orthird electronic device 130 a, 130 b, as shown in FIGS. 1, 2 and 3),having the base plate 125 of the cold plate in direct contact with asurface of the second electronic device. In this way, heat may transferfrom the surface of the electronic device, through the thermal interfaceprovided by the base plate 1245, to the liquid coolant flowing withinthe cold plate.

All of the features disclosed herein may be combined in any combination,except combinations where at least some of such features and/or stepsare mutually exclusive. In particular, the preferred features of theinvention are applicable to all aspects of the invention and may be usedin any combination. Likewise, features described in non-essentialcombinations may be used separately (not in combination).

In particular, although FIGS. 1 to 4B above discuss a particularpreferred embodiment of the invention in which one or more weir isincorporated within the first cooling circulatory system, and one ormore cold plate is incorporated within the second cooling circulatoryarrangement, these components are not necessarily required. Inparticular, in its most general form the concept described herein is theuse of a first and second cooperating cooling circulatory system (orcooling loop), whereby both the first and second cooling circulatoryarrangement each cool different electronic devices within an electronicmodule, and further wherein the second cooling circulatory arrangementis used to cool the first cooling circulatory system.

This most general concept is depicted in FIGS. 13A and 13B. The examplesof FIG. 13A and FIG. 13B illustrate the following first common feature:a first cooling circulatory system (dotted line) arranged to circulate afirst liquid coolant between a first electronic device 1320 of aplurality of electronic devices and a heat exchanger 1330. In thespecific example of FIGS. 13A and 13B, a pump 1325 is shown as formingpart of the first cooling circulatory system, although this is notessential, and circulation could take place by other mechanisms (such asconvection). In the first cooling circulatory system of FIGS. 13A and13B, the first electronic device 1320 is thermally coupled to thecirculating first liquid coolant (when the system is in use), such thatheat is transferred from the first electronic device 1320 to the firstliquid coolant.

The examples of FIG. 13A and FIG. 13B illustrate the following secondcommon feature: a second cooling circulatory arrangement (bold line) isarranged to circulate a second liquid coolant between a second 1335 (andthird 1340, in parallel) electronic device of the plurality ofelectronic devices and the heat exchanger 1330. The second and thirdelectronic devices 1335, 1340 are thermally coupled to the second liquidcoolant circulating in the second cooling circulatory arrangement (whenthe system is in use), such that heat is transferred from the second andthird electronic devices 1335, 1340 to the second liquid coolant.

The examples of FIG. 13A and FIG. 13B illustrate the following thirdcommon feature: the first cooling circulatory arrangement and the secondcooling circulatory arrangement are thermally coupled via the heatexchanger 1330. In other words, as the first liquid coolant and thesecond liquid coolant each pass through the heat exchanger 1330, heat istransferred from the first liquid coolant to the second liquid coolantvia a thermal interface within the heat exchanger 1330.

In addition, FIG. 13A depicts a cooling system as part of the secondcooling circulatory arrangement, and FIG. 13B depicts the second coolingcirculatory arrangement being connected to a mains water and drainage.This is equivalent to these portions of the second cooling circulatoryarrangement discussed above with respect to FIGS. 4A and 4B.

Accordingly, a system is shown in which heat is removed and transferredaway from the first liquid coolant in the first cooling circulatoryarrangement by heat transfer to the second liquid coolant in the secondcooling circulatory arrangement. The heat transfer primarily takes placeat a dedicated heat exchanger 1330 which provide efficient removal ofheat from the first cooling circulatory arrangement. The describedsystem allows for efficient cooling of both specific components of theelectronic module (by the second cooling circulatory arrangement) butalso cooling of the overall volume of the electronic module (by thefirst cooling circulatory arrangement). The described system isfurthermore compact, so as to be able to be used within industrystandard server blades or chassis, and can be more easily installed incommonplace server racks, as a result of the closed loop first coolingcirculatory arrangement and quick disconnect mechanisms for the secondcooling circulatory arrangement.

As such the described system describes an efficient and adaptable systemmaking use of a hybrid arrangement of targeted cooling and cooling of avolume within an electronic module.

Although a specific configuration of two cold plates within anelectronic module is shown in FIGS. 1 to 4B above, it will be understoodthat various cold plate configurations are possible. Moreover, differentconfigurations of the cold plates, and the connection of the associatecirculatory system, to a rack (or server) can provide a number ofbenefits. There is now described a variety of systems for cooling anelectronic module configured for installation into a rack, as well asmethods of installation of a liquid cooled system for an electronicmodule. Each of the described configurations incorporates at least onecold plate (as described above with respect to FIGS. 12A, 12B and 12C),and could be combined with the above described embodiments, inparticular with respect to incorporation within the second coolingcirculatory arrangement.

FIG. 14 shows a plan view of an electronic module (or server module) inwhich a cold plate assembly 1401 is arranged. The electronic module isof a type for mounting in an existing rack server system. The cold plateassembly 1401 may be installed in place of a previous air-cooled heatsink module. Advantageously, the cold plate assembly may be installed inthe same footprint as the air-cooled heat sink module. Accordingly, thecold plate assembly has the same or similar footprint or dimensions tocommonly used air-cooled heat sink modules.

The cold plate assembly 1401 is mounted within the housing or chassis(or server chassis) 1420 of the electronic module 1410. The serverchassis 1420 may be for instance a 1RU server chassis, adhering toestablished industry standards. The base plate of the cold plate module1401 is arranged to be thermally coupled to an electronic device mountedwithin the housing. The cold plate module may be mounted directed to thesubstrate (or printed circuit board) 1425 forming the base of the modulehousing 1420. Input conduit 1430 is coupled to the inlet port 1435 ofthe cold plate. Output conduit 1440 is connected to outlet port 1445 ofthe cold plate. Besides the novel features associated with the coldplate assembly, the illustrated electronic module also shows variousother features that would be typical in a standard electronic module, inparticular: in/out ports 1450 (including USB, QSFP and Ethernet ports)for electrical or data connections (at the front of the electronicmodule, if viewed when the module is mounted in a rack), and fans 1455for air cooling the cavity within the module housing (at the rear of theelectronic module, if viewed when the module is mounted in a rack). Themodule housing may house various electronic components or devices 1460(RAM chips, etc).

In use, the cold plate assembly 1401 and inlet 1430 and outlet 1440conduits are connected within a cooling loop, together with a coolingsystem and pump (not shown in FIG. 14). Liquid coolant is circulatedaround the cooling loop. Specifically, liquid coolant is passed throughthe input conduit 1430 into the cold plate assembly 1401. The liquidcoolant passes through the channel in the housing of the cold plateassembly, as discussed above in relation to FIGS. 12A, 12B and 12C, inorder to receive heat transferred to the liquid coolant from anelectronic device thermally coupled to the cold plate housing. As such,the liquid coolant exiting the cold plate assembly 1401 has a highertemperature than liquid coolant entering the cold plate assembly 1401.

Liquid coolant exiting the cold plate assembly 1401 is received throughthe outlet conduit 1440. The liquid coolant is then transferred throughthe output conduit 1440 to the cooling system and pump (not shown, andwhich may comprise a combined unit). At the cooling system, a heatexchanger is provided to transfer heat from the liquid coolant to asecond cooling medium. The second cooling medium may be air, forinstance, or may be a second cooling loop through which a second liquidcoolant is circulating. The cooling system reduces the temperature of,or cools, the liquid coolant. The liquid coolant cooling loop may beformed as part of a large scale cooling facility (such as a buildingcooling water loop). In an alternative example, the liquid cooling loopmay be formed as a smaller loop that is local to the server module, orto the rack to which the server module is connected. The liquid coolingloop can be arranged according to local provision and requirements.

After cooling at the cooling system, the cooled liquid coolant is nextcirculated further around the cooling loop though the input conduit 1430to the cold plate module 1401, thereby completing the cooling loop.

As will be understood by the person skilled in the art, various othercomponents may be included within the cooling loop (such as switches,valves, manifolds, or additional pumps). Additional cold plateassemblies may also be comprised within the cooling loop, as describedbelow. The described cooling loop is provided for illustrative purposes,and is not considered to be limiting.

FIG. 15 describes a further example of an electronic module (or servermodule) 1510, in which two cold plate assemblies 1401 a, 1401 b arearranged. The same example is depicted in FIG. 16, in plan view. Theelectronic module 1510 comprises a module housing 1420 with substrate1425, in/out ports 1450, and fans 1455. Electrical components includingcomputer RAM 1460 and capacitors 1565 are housed within the modulehousing 1520.

A first inlet 1435 a and a first outlet 1445 a port is provided in thelid of the first cold plate assembly 1401 a. Similarly, a second inlet1435 b and a second outlet 1445 b port is provided in the lid of thesecond cold plate assembly 1401 b. A first and second input conduit 1430a, 1430 b is attached to the respective first and second inlet ports1435 a, 1435 b, and a first and second output conduit 1440 a, 1440 b isattached to the respective first and second outlet ports 1445 a, 1445 b.The first 1401 a and second 1401 b cold plate assemblies are arrangedwithin the cooling loop in a parallel configuration. As such a singlesupply conduit 1550 is connected to both of the first 1430 a and second1430 b inlet conduits. Similarly, a single receiving conduit 1555 isconnected to both the first 1440 a and second 1440 b output conduits.The supply conduit 1550 could be considered to ‘split’ into the parallelcanals provided by the first and second input conduits, for instance.

At least one coupling 1570 a, 1570 b is provided for connection of thesupply conduit 1550 to the first 1430 a and second 1430 b inletconduits, and/or for the connection of the first 1440 a and second 1440b output conduit to the receiving conduit 1555. In this example, thecouplings 1570 a, 1570 b reside outside of the module housing 1420. Assuch, first 1430 a and second 1430 b input conduits and first 1440 a andsecond 1440 b output conduits are arranged to pass through an opening1480 in the wall of the housing 1420, to transfer the liquid coolantbetween the inside and outside of the cavity within the electronicmodule. Advantageously, the opening 1480 is an aperture provided at therear of a typical server module, for instance an existing PCIE cardslot, as illustrated in FIG. 15.

As shown in FIGS. 3 and 4, the input 1430 a, 1430 b and output 1440 a,1440 b conduits are arranged around the electrical components 1565, 1460arranged within the module housing. Beneficially, the conduits areprovided by flexible tubing or pipes, having a narrow diameter (or boresize size). As such, the conduits can more easily be fitted around theelectrical components. Furthermore, the tubing can be arranged to passthrough typically provided, or existing apertures in the wall of themodule housing (such as those for an electrical or data port, or forcomponents such as data cards).

The described characteristics of the cold plate assembly 1401 a, 1401 bare particularly useful when the cold plate assembly is retrofitted, forinstance in place of an air-cooled heat sink, as the electricalcomponents are already installed in the module housing, and it isdesirable that existing components are not moved or relocated. As such,when retrofitting the cold plate assembly, the fitter can advantageouslyfit the cold plate module within the foot print of a previous coolingdevice (as a result of the dimensions of the cold plate assembly), andfurther, the flexible, narrow tubing used for the conduits together withswivel nozzle at the ports to each cold plate assembly, can allow theinstaller to assess and apply the most natural route for the tubingthrough the server chassis and around existing components.

In use, liquid coolant is passed through the supply conduit 1550 to thefirst 1430 a and the second 1430 b input conduits, via the coupling 1570a. From here, the first 1401 a and second 1401 b cold plate assemblyform part of separate, first and second parallel branches of the coolingloop. The liquid coolant is circulated through each of the first 1430 aand second 1430 b input conduits in parallel, to be input through theinlet port 1435 a, 1435 b of each of the first and second cold platesfor the respective first and second input conduit. The liquid coolantthen passes through the inner channel of each of the cold plates,receiving heat transferred from the thermally coupled electronicdevices, before exiting the cold plate through the outlet ports 1445 a,1445 b. The liquid coolant is then transferred to each of the first 1440a and second 1440 b output conduits, to join the receiving conduit 1555at the coupling 1570 b.

In use, the supply conduit 1550 and receiving conduit 1555 are connectedto a cooling system and/or pump in order to complete the cooling loop,as described above with respect to the example of FIG. 14. The coolingsystem acts to remove heat from the liquid coolant, therefore removingheat originally transferred from the electrical components out of thecooling loop.

FIG. 17 shows a plan view of a further example of an electronic module1710 including four cold plate assemblies. The configuration of thefirst 1401 a and second 1401 b cold plate assembly, and further of thethird 1401 c and fourth 1401 d cold plate assembly is each similar tothe arrangement of the first and second cold plate assembly shown inFIGS. 15 and 16.

More particularly, a first 1401 a and second 1401 b cold plate assemblyare connected within a liquid coolant loop in parallel, in a similararrangement to that described with respect to FIGS. 15 and 16 (althoughthe position of the first 1401 a and second 1401 b cold plate assemblieson the substrate 1425 of the electronic module are different that thoseof FIGS. 15 and 16, in order to optimise the arrangement of the tubingforming the first and second input 1430 a, 1430 b and output 1440 a,1440 b conduits). In this example, the supply and receiving conduitsrepresent a first supply 1550 a and a first receiving 1555 a conduit.

A third 1401 c and fourth 1401 d cold plates are also provided. Thesystem of the third and fourth cold plate, the third 1430 c and fourth1430 d input conduits, and third 1440 c and fourth 1440 d outputconduits are equivalent to those of the first and second cold plate, thefirst and second input conduits, and first and second output conduits,respectively. As such, the third and fourth cold plate assemblies 1401c, 1401 d are connected within a liquid coolant loop in parallel. Thethird 1430 c and fourth 1430 d input conduits are coupled to, andreceive liquid coolant from, a second supply conduit 1550 b. Similarly,the third 1440 c and fourth 1440 d output conduits are coupled to, andtransfer liquid coolant to, a second receiving conduit 1555 b.

The first and second supply conduit 1550 a,b shown in FIG. 17 may beconnected in parallel within the liquid cooling loop. In other words, asingle output from the cooling system of the cooling loop may be splitto each connect to the first 1550 a and second 1550 b supply conduit.Similarly, the first 1555 a and second 1555 b receiving conduit can beconnected in parallel, such that the first and second receiving conduitjoin to pass to the input of the cooling system as a single entity.

In an alternative example, the first 1550 a and the second 1550 b supplyconduit and the first 1555 a and the second 1555 b receiving conduit maybe connected in separate, respective first and second cooling loops,each having a cooling system. This may provide a greater cooling powerthan the arrangement discussed above in which first and the secondsupply conduit 1550 a,b and the first and the second receiving conduit1555 a,b are arranged in series. However, it also requires a morecomplex infrastructure, and may be more costly. The specificconfiguration for the cooling loop connected to the apparatusillustrated in the example of FIG. 17 (and other examples depictedherein) may be selected according to the cooling requirements andinfrastructure of a particular server facility.

FIG. 18 depicts an electronic module 1810 in which a first 1401 a and asecond 1401 b cold plate assembly is mounted. In this example, the firstand second cold plates are arranged in the cooling loop in parallel,having first 1430 a and second 1430 b input conduits, first 1440 a andsecond 1440 b output conduits, couplings 1570 a, 1570 b and source 1550and receiving 1555 conduits arranged in much the same manner as FIGS. 15and 16. However, in this example, the couplings 1570 a, 1570 b at whichthe first 1430 a and second 1430 b input conduits are coupled to thesupply conduit 1555, and at which the first 1440 a and the second 1440 boutput conduits are coupled to the receiving conduit 1555, are arrangedwithin the module housing 1420. In this case, the source 1550 andreceiving conduits 1555 are arranged to pass through an opening in themodule housing.

Compared to the arrangement in FIGS. 15 and 16, this configurationbenefits that fewer pipes or tubes must be fed-though the wall of themodule housing. However, in order to maintain a generally efficient flowof liquid coolant though the first and second cooling assembly, thesource and receiving conduits will typically be provided by tubes havinga larger bore size or diameter than each of the first and second inputand output conduits. Therefore, in some scenarios, placing the couplingwithin the module housing (as shown in the example of FIGS. 15 and 16)may be preferable, as it allows for greater flexibility in the placementof the feedthrough of each conduit.

FIG. 19 depicts a plan view of an electronic module 1910 housing a first1401 a and second 1401 b cold plate. However, compared to previousexamples, the first 1401 a and second 1401 b cold plate are arranged inseries within the cooling loop. As such, although a first input conduit1430 a is connected to a first inlet port 1435 a of the first coldplate, and a second output conduit 1440 b is connected to the outletport 1440 b of the second cold plate, as before, in this case the firstoutlet port 1445 a is coupled directly to the second inlet port 1435 bvia a link conduit 1915. The first input conduit 1430 a and the secondoutput conduit 1440 b are arranged to pass through an opening in thewall of the module housing 1420. The first input conduit 1430 a and thesecond output conduit 1440 b are subsequently connected to the coolingloop as previously described.

In use, the liquid coolant is transferred through the first inputconduit 1430 a to be received at the inlet port 1435 a of the first coldplate 1401 a. The liquid coolant passes through the channel of the firstcold plate, receiving heat transferred from an electronic devicethermally coupled thereto. The cooling liquid then passes out of thefirst cold plate via the first outlet port 1445 a, to be transferredthought link conduit 1915 to the inlet port 1435 a of the second coldplate 1401 b. The liquid coolant then further passes through the channelof the second cold plate, receiving heat transferred from an electronicdevice thermally coupled to the second cold plate. The liquid coolant isthen passed out of the second cold plate via the second outlet port 1445b, to the second output conduit 1440 b. The liquid coolant is furthercirculated towards a cooling system within the cooling loop.

In the series connected cold plates of FIG. 19, the liquid coolant has afirst temperature before entry to the first cold plate (i.e. in thefirst input conduit 1430 a), a second temperature after exit from thefirst cold plate and before entry to the second cold plate (i.e. in thelink conduit 1915), and a third temperature after exiting the secondcold plate (i.e. in the second output conduit 1440 b). The firsttemperature will be lower than the second temperature, and the secondtemperature will be lower than the third temperature. In other words,the liquid coolant becomes progressively hotter as it passes throughadditional cold plates, in view of the heat transferred within thechannel of each cold plate.

The efficiency of transfer of heat from an electronic device to theliquid coolant in a channel of a cold plate is dependent on thetemperature gradient (or the differential temperature) between theelectronic device and the liquid coolant. Therefore, where thetemperature of the liquid coolant is higher, and so closer to theoperating temperature of the electronic device, the heat transfer fromthe electronic device to the liquid coolant (i.e. the cooling) maybecome less efficient. As such, in the series configuration of the coldplates shown in FIG. 19, an electronic device thermally coupled to thefirst cold plate 1401 a may be more efficiently cooled than anelectronic device thermally coupled to the second cold plate 1401 b.However, the series configuration of cold plate modules is more compactand requires fewer pipes and tubes to act as conduits. Moreover, themaximum possible cooling efficiency may not be required for everyelectronic device within a given electronic module. Therefore, thesequence of cold plate modules in the series configuration may be chosento provide sufficient cooling to each electronic device to be cooled.

The difference in efficiency of the cooling between the first and secondcold plates described in FIG. 19 can be effectively overcome byprovision of each of the cold plate in parallel within the cooling loopas shown in the examples of FIGS. 15, 16, 17 and 18, for instance. Inthe parallel configuration, the temperature of the liquid coolantcirculating through each cold plate is the same. Therefore, each coldplate, attached within each parallel branch of the cooling loop, has thesame potential cooling efficiency for an electronic device operating ata given temperature.

Nevertheless, the parallel configuration described may be morecumbersome to install within a server blade (especially where thecooling system is retrofitted), as a larger number of conduits (pipes ortubes) and couplings are required to achieve a parallel configuration,particularly as more cold plates are added within a specific electronicmodule or server blade. A greater number of conduits may need to be fedthrough existing apertures or opening in the wall of the module housing,in order to service the cold plate modules arranged in parallel.Therefore, in some cases, a series arrangement of cold plate modules maybe preferable to a parallel configuration of cold plate modules,dependent on the specific space considerations and cooling requirementsof a particular electronic module.

With this in mind, FIG. 20 depicts a further example of a liquid coolingsystem installed within an electronic module 2010. In this example, twoparallel branches of a cooling loop circulate though cold plates mountedwithin the electronic module. On the first branch of a cooling loop afirst, a second and a third cold plate are arranged in series. On asecond branch of the cooling loop, a fourth and a fifth cold plate arearranged in series. The footprint and volume of each of the cold plateson the first branch (the first, second and third cold plate) is smallerthan the footprint and the volume of each of the cold plates on thesecond branch (the fourth and fifth cold plate). The cooling power ofthe larger cold plates (the fourth and fifth cold plate) may be greaterthan that of the smaller cold plates (the first, second and third coldplate), as they provide a larger surface area for transfer of heat froman electronic device to the cooling liquid. However, the cooling powerwill be in some part limited by the rate of flow of liquid coolantthrough each branch of the cooling loop (which may in turn be determinedby the bore size or diameter of the pipes forming the conduits forconnection of the cold plates).

The specific liquid cooling system of FIG. 20 comprises first 1401 a,second 1401 b, third 1401 c, fourth 1401 d and fifth 1401 e cold plates.Each cold plate has respective inlet and outlet ports. A supply conduit350 provides a passage for liquid coolant received from a coolingsystem, prior to splitting into the two parallel branches of the coolingloop. In the first branch from the supply conduit, a first input conduit1430 a is arranged to transfer liquid coolant to the inlet port 1435 aof the first cold plate, a first link conduit 1915 a is arranged totransfer liquid coolant from the first outlet port 1445 a of the firstcold plate to the second inlet port 1435 b of the second cold plate, asecond link conduit 1915 b is arranged to transfer liquid coolant fromthe second outlet port 1445 b of the second cold plate to the thirdinlet port 1435 a of the third cold plate, and a first output conduit1440 a is arranged to transfer liquid conduit from the outlet port 1445c of the third cold plate to a coupling 1570 b with the receivingconduit 1555. In the second branch from the supply conduit (arranged inparallel to the first branch), a second input conduit 1430 b is arrangedto transfer liquid coolant to the inlet port 1435 d of the fourth coldplate, a third link conduit 1915 c is arranged to transfer liquidcoolant from the outlet port 1445 d of the fourth cold plate to theinlet port 1435 e of the fifth cold plate, and a second output conduit1440 b is arranged to transfer liquid coolant from the outlet port 1445e of the fifth cold plate to a coupling with the receiving conduit 1555.

In use, liquid coolant is circulated around each branch of the coolingloop. Comparatively cooler liquid coolant is supplied by the supplyconduit to the input conduit of each branch, with comparatively hotterliquid coolant being received at the receiving conduit from each of theoutput conduits of each branch. A pump and a cooling system (asdescribed above) can be arranged between the receiving conduit and thesupply conduit within the cooling loop, in order to circulate the liquidcoolant and to transfer heat out of the liquid coolant.

As will be understood, any number of configurations for the cold platesand cooling loop could be arranged based on the examples outlined. Morethan two parallel branches of the cooling loop could be arranged withinan electronic module, and any number of cold plates could be arranged inseries on each branch of the cooling loop. The specific arrangement willbe selected according to the cooling requirements of a particularelectronic module, and in view of the space and configurationrestrictions of the electronic module (especially when retrofitting thedescribed cooling system into an existing electronic module). Thespecific arrangement may be selected (for instance, by an installer) tooptimise the efficient cooling of the electronic module in view of theserequirements and restraints.

FIG. 21 shows a plan view of a further example for the cooling systeminstalled within an electronic module 2110. The electronic module 2110is shown mounted within a rack 2100. In this example, a first 1401 a andsecond 1401 b cold plate are arranged in parallel within a cooling loop.Within the boundary of the module housing 1420, the configuration of afirst 1430 a and second 1430 b input conduit, a first 1440 a and second1440 b output conduit and the first and second cold plates are the sameas described above with respect to the example of FIGS. 15 and 16.However, in this example, a first 2120 a and second 2120 b manifold areprovided external to the electronic module. The first manifold 2120 a iscoupled to the first 1430 a and second 1430 b input conduits, and thesecond manifold 2120 b is coupled to the first 1440 a and second 1440 boutput conduits. It will be understood that a source and receivingconduit will be connected to the first and second manifold,respectively, each having a similar function within the cooling loop asthe source and receiving conduit described above with respect to FIGS.15 and 16. In FIG. 21, the source and receiving conduits are notvisible, being connected to the manifolds perpendicular to the directionof flow of the liquid coolant in the input and output conduits (i.e. theflow of the liquid coolant through the source and receiving conduitswill be into/out of the plane of the electronic module in FIG. 21).

Each manifold provides a unit having a passage or channel for supply ofliquid coolant to the conduits. The passage or channel within themanifold may have a wider diameter or bore size than the conduits, inorder to avoid limitation to the flow rate of the liquid coolant throughmanifold. The manifold unit made be rigid, allowing easier fixture tothe electronic module or rack than a simple coupling between flexibletubing (as show with respect to couplings 1570 a, 1570 b in FIGS. 15 and16, for instance). Furthermore, provision of a rigid manifold as a pointof coupling of the input and output conduits to the respective supplyand receiving conduits reduces strain on the coupling, and so helps toprevent leaks of liquid coolant, or reduction of pressure in the liquidcooling loop. Finally, use of the described manifold aids in theinstallation and fitting of the cooling system, as it allows fordecouplable connection between conduits, and so flexibility in thearrangement of the various components of the cooling system.

The manifold may be supported by a bracket 2130, which in the example ofFIG. 21 is connected to the rear of the module housing. In analternative example, the bracket may be connected to the rack.

Each of the first and second input conduits are connected or coupled tothe first manifold using connectors 2125 a, 2125 b, and morespecifically blind mate connectors, although other types of connectorcould be used. Blind mate connectors have a mating action that is‘push-fit’, in other words by sliding or snapping the connector pluginto the socket. As such, the connectors are easier to fit, without theuse of tools such as wrenches. Moreover, no torque needs to be appliedto the tubing or pipes providing the conduit when fitting (which allowsfor greater control of the arrangement of the conduits within thecooling system). Furthermore, the blind mate connectors haveself-aligning features which allows resilience to a small misalignmentwhen mating. Thus, this type of connector provide greater ease offitting the cooling system, especially when installed into existingelectronic modules in place of an air cooled heat sink system.

FIG. 22 depicts a plan view of a further example of the cooling systeminstalled within an electronic module 2200. This example also includes amanifold, but in a further configuration compared to that shown in FIG.21. The cooling system of FIG. 22 comprises first 1401 a and second 1401b cold plates, arranged in parallel within the cooling loop. Thearrangement of the first 1401 a and second 1401 b cold plate, the first1430 a and second 1430 b input conduit, the first 1440 a and second 1440b output conduit, and the source 1550 and receiving 1555 conduits areidentical to those described above with respect to the examples of FIGS.15 and 16.

In the example of FIG. 22, the source 1550 and the receiving 1555conduits are each connected to a respective first 2120 a and second 2120b manifold. The manifolds may connect the source and receiving conduitsinto the cooling loop, as previously described. Similar to the manifolddiscussed above with respect to FIG. 21, the manifolds 2120 a, 2120 bare units or elements having a passage or channel for supply of liquidcoolant to the conduits. They may be rigid, and provide the variousbenefits outlined above.

The supply and receiving conduits may be connected to the manifolds viaconnectors 2225 a, 2225 b. Said connectors may be manual connectors, forinstance, which require manual connection during installation (or manualdisconnection in order to remove the electronic module from the rack).Such connectors are relatively simple to install, and less complex thanother types of connector. Therefore, this type of drip-free, manualconnector may be particularly useful when retrofitting the describedcooling system to an existing electronic module.

In the example of FIG. 22, the first 2120 a and second 2120 b manifoldare supported by the rack 2100. More particularly, the first 2120 a andsecond 2120 b manifold are mounted within a duct (or passageway, orcavity) 2101 provided in a typical rack for housing electrical and datacables. The duct 2101 provides a cavity adjacent to the rear of theelectronic module 2100, such that the first 2120 a and second 2120 bmanifold mounted within are housed in a compact and secure fashion.Beneficially, this arrangement for housing the manifolds 2120 a, 2120 bdoes not necessarily require any additional brackets or infrastructureto be provided for support of the manifold, and still securely androbustly affixes the manifolds in place.

It is noted that the manifolds described above with respect to FIGS. 21and 22 are shown as separate manifold units for each of the input andoutlet portions of the cooling loop. It will be understood that bothinput and outlet portions may be serviced by a single manifold unit withappropriate partition therein. Nevertheless, separate manifolds for eachof the input and outlet portions may be beneficial to provide additionalflexibility in the arrangement of the manifolds, especially whenretro-fitting the cooling system to an existing server blade.

A cooling loop, of which the described cold plate apparatus form a part,has been described above. FIG. 23 depicts a schematic view of a specificconfiguration of a full cooling loop, by way of example. FIG. 23 shows:an electronic module or server chassis 2300; a first 2320 a and a second2320 b cold plate in accordance with the present disclosure; awater-based cooling loop 2330 (which is the “cooling loop” circulatingthrough the cold plates, as described with reference to the examplesabove); a first and second manifold 2340 a, 2340 b; and a coolingdistribution unit (CDU) 2350 comprising a heat exchanger 2360 (orcooling system) for transfer of heat from the liquid coolant in liquidcooling loop 2330 to a heat sink 2380. A facility level pump 2370 isused to distribute coolant to all the cold plates 2320 a, 2320 b in thecooling loop 2330. The electronic module or server chassis 2300 may bemounted within a rack 2310, with the manifolds 2340 a, 2340 b beingmounted to the rack and also being used to direct coolant to and fromother servers (not shown). The manifold may be mounted within a duct ofthe rack for storage and passage of electronic and data cables, asillustrated in FIG. 22.

It will be understood that although the pump and heat exchanger areshown within a unit (a CDU) in FIG. 23, instead a separate coolingsystem (or heat exchanger) and pump could be implemented within coolingloop 2330.

FIG. 24 shows an alternative example for the cooling loop of which oneor more cold plates described in the examples above may form a part.FIG. 24 depicts a schematic view of the electronic module 2400 mountedin a rack 2410, including: a first 2420 a and a second 2420 b cold platein accordance with the present disclosure; a first cooling loop 2430(which is the “cooling loop” circulating through the cold plates, asdescribed with reference to the examples above); a cooling systemcomprising heat exchanger 2465, a first pump 2475, a second cooling loop2435, a first and a second manifold 2440 a, 2440 b; and a coolingdistribution unit (CDU) 2450 comprising a heat exchanger 2460 fortransfer of heat to a heat sink 2480 and a facility level pump 2470.

In this example, the first cooling loop 2430 (or “cooling loop”, asdescribed with respect to FIGS. 13 to 22 above) circulates through thecold plates 2420 a, 2420 b, and is entirely contained within the chassisof the electronic module 2400. The first cooling loop 2430 is circulatedby pump 2475, which is also housed within the chassis. The first coolingloop 2430 is connected to a cooling system which here comprises a heatexchanger 2465 for transfer of heat from the liquid coolant circulatingin the first cooling loop to a cooling medium (for instance water, oranother liquid coolant) circulating in the second cooling loop 2435. Inthis way, heat is removed from the first cooling loop 2430. The coolingsystem in this example is also housed within the chassis of theelectronic module 2400.

Second cooling loop 2435 circulates between the heat exchanger and thecooling distribution unit. Circulation may be effected by pump 2470within the CDU. The second cooling loop 2435 may be a facility levelcooling loop (i.e. a building water cooling loop). Alternatively, itcould be local to the electronic module, to the rack of electronicmodules, to the server room in which the modules are housed, forinstance. The second cooling loop may pass through manifolds 2440 a,2440 b, which are mounted to the rack 2410. In an alternative, manifoldscould be mounted to the electronic module.

At the CDU, heat is transferred from the second cooling loop 2435 viaheat exchanger 2460. The heat is transferred to a heat sink 2480. Itwill be understood that the heat sink 2480 could represent an air-cooledheat sink. Alternatively, the heat sink 2480 could form part of afurther (third) cooling loop, through which liquid coolant iscirculated. This may be appropriate where the second cooling loop islocal to a specific rack of electronic modules, and the third coolingloop is a facility level cooling loop (such as a building water coolingloop).

As will be understood by the person skilled in the art, electronicmodules such as those discussed above with respect to FIGS. 14 to 24 areconfigured to be fitted into racks. In particular, each rack can house aplurality of electronic modules. In this way a bank of electronicmodules are formed, for example each providing a server within a serverbank. The cooling systems, discussed above with respect to individualelectronic modules, therefore form part of a larger cooling system thatmay serve a plurality of electronic modules in a rack, or in one or moreracks.

FIG. 25 shows a schematic view of an example cooling system comprising aplurality of electronic modules, each with a cooling system installed.FIG. 25 shows a server chassis or electronic module 2500 mounted withina rack 2510. Multiple racks may be provided, each of which can house upto 42 electronic modules 2500. In this example, a water-based coolingloop 2530 is provided across all electronic modules 2500. The coolingloop 2530 is cooled using a single coolant distribution unit (CDU) 2550.The CDU 2550 includes a heat exchanger 2560 and a facility-level pump2570, with heat being transferred to a heat sink 2580. The heat sink2580 may itself be air cooled. Alternatively, the heat sink 2580 mayform part of a second coolant loop (not shown). In either case, the heatexchanger assists in transfer of heat from the liquid coolantcirculating in the cooling loop 2530. Thus, a single pump 2570 can beused in a system with a plurality (even hundreds) of electronic modules2500.

It is noted that each of the server chassis or electronic modules 2500shown in FIG. 25 are configured with a cooling loop according to serverchassis or electronic module 2300 shown in greater detail in FIG. 23.However, it will be understood that each of the electronic modules 2500in FIG. 25 could instead have a configuration according to theelectronic module 2400 shown in FIG. 24. In particular, each electronicmodule 2500 of FIG. 25 could include a cooling system (comprising atleast a heat exchanger 2465, to transfer heat to a second cooling loop2435) and a pump 2475 within the body of the electronic module or serverchassis, according to the illustrated electronic module 2400 of FIG. 24.

In summary, a range of benefits are provided by embodiments inaccordance with the disclosure. These are particularly advantageous in acold plate for cooling electronics cooling in dense applications (forexample 1U sized servers in multiple racks) and ultra-denseapplications. A particular benefit is provided by the adaptability andconfiguration of the described cooling system, which makes itparticularly appropriate to retrofitting to existing electronic modulesor server rack systems. Furthermore, the described system can be appliedto standard or typical server chassis or server racks, without complexcustomisation.

Moreover, any of the described configurations for the cold plates inFIGS. 14 to 25 could be applied to the first and second coolingcirculatory arrangements, as discussed above with respect to FIGS. 1 to4B. In particular, the described first cooling circulatory arrangementcould be combined with any of the described cold plate configurationsdiscussed with reference to FIGS. 14 to 25 (and implemented as thesecond cooling circulatory arrangement), with the addition of a suitableheat exchanger 170 for transfer of heat between the first and the secondcooling circulatory arrangements.

A number of combinations of the various described embodiments could beenvisaged by the skilled person. All of the features disclosed hereinmay be combined in any combination, except combinations where at leastsome of such features and/or steps are mutually exclusive. Inparticular, the preferred features of the invention are applicable toall aspects of the invention and may be used in any combination.Likewise, features described in non-essential combinations may be usedseparately (not in combination).

1. A system for cooling a plurality of electronic devices housed in ahousing of an electronic module, the system comprising: a first coolingcirculatory arrangement, configured to circulate a first liquid coolantbetween a first electronic device of the plurality of electronic devicesand a heat exchanger, the first electronic device being thermallycoupled to the first liquid coolant such that heat is transferred fromthe first electronic device to the first liquid coolant; and a secondcooling circulatory arrangement, configured to circulate a second liquidcoolant between a second electronic device of the plurality ofelectronic devices and the heat exchanger, the second electronic devicebeing thermally coupled to the second liquid coolant such that heat istransferred from the second electronic device to the second liquidcoolant; wherein the first cooling circulatory arrangement and thesecond cooling circulatory arrangement are thermally coupled at leastvia the heat exchanger, such that heat is transferred from the firstliquid coolant to the second liquid coolant via the heat exchanger. 2.The system of claim 1, wherein the second cooling circulatoryarrangement further comprises a cooling system, wherein the secondcooling circulatory arrangement is configured to circulate the secondliquid coolant between the second electronic device of the plurality ofelectronic devices, the heat exchanger and the cooling system, whereinheat is removed from the second liquid coolant by the cooling system. 3.The system of claim 1, wherein the second cooling circulatoryarrangement is connected to a second liquid coolant supply, wherein thesecond cooling circulatory arrangement is configured to circulate thesecond liquid coolant received from the second liquid coolant supplybetween the second electronic device of the plurality of electronicdevices and the heat exchanger, and to be returned to the second liquidcoolant supply.
 4. The system of claim 1, wherein the heat exchangercomprises at least a first and a second chamber separated by a thermalinterface, wherein the heat exchanger is configured for flow of thefirst liquid coolant through at least the first chamber, and flow of thesecond liquid coolant through at least the second chamber, such thatheat is transferred from the first liquid coolant to the second liquidcoolant through the thermal interface.
 5. The system of claim 1, whereinthe heat exchanger is arranged within the housing of the electronicmodule.
 6. The system of claim 1, wherein the housing of the electronicmodule contains the first liquid coolant, and wherein the firstelectronic device is at least partially immersed in the first liquidcoolant.
 7. The system of claim 6, wherein the first cooling circulatoryarrangement further comprises a weir, the weir comprising: a base and aretaining wall extending from the base, the base and retaining walldefining a volume for holding some of the first liquid coolant; aninlet, through which the first liquid coolant flows into the volume;wherein the flow of sufficient first liquid coolant through the inletinto the volume causes the first liquid coolant to overflow theretaining wall and collect with first liquid coolant contained in thehousing of the electronic module and exterior the weir.
 8. (canceled) 9.The system of claim 7, wherein the weir further comprises projectionsextending from the base and/or retaining wall within the volume of theweir.
 10. The system of claim 7, wherein the weir is coupled to asurface of the first electronic device, to act as a heat sink.
 11. Thesystem of claim 1, wherein the first cooling circulatory arrangementfurther comprises: a pump configured to circulate the first liquidcoolant around the first cooling circulatory arrangement.
 12. The systemof claim 11, wherein the first cooling circulatory arrangement furthercomprises a pump inlet, arranged to receive first liquid coolantcontained in the housing of the electronic module and exterior the weir.13. (canceled)
 14. The system of claim 1, the second cooling circulatoryarrangement further comprising a cooling module configured to thermallycouple the second electronic device to the second liquid coolant. 15.The system of claim 14, wherein the cooling module comprises a coldplate, the cold plate comprising: a cold plate housing, a surface of thecold plate housing being arranged to provide a thermal interface forcooling the second electronic device which is thermally coupled thereto;and at least one channel within the cold plate housing and proximate tothe surface of the cold plate housing, the at least one channel arrangedfor the second liquid coolant to flow therethough such that heatreceived from the second electronic device through the surface of thecold plate housing is transferred to the second liquid coolant.
 16. Thesystem of claim 15, the second cooling circulatory arrangement furthercomprising a plurality of conduits arranged to transport the secondliquid coolant between the cold plate, the heat exchanger and thecooling system.
 17. A method for cooling a plurality of electronicdevices housed in a housing of an electronic module, the methodcomprising: circulating a first liquid coolant around a first coolingcirculatory arrangement, comprising circulating a first liquid coolantbetween a first electronic device of the plurality of electronic devicesand a heat exchanger, the first electronic device being thermallycoupled to the first liquid coolant such that heat is transferred fromthe first electronic device to the first liquid coolant; and circulatinga second liquid coolant around a second cooling circulatory arrangement,comprising circulating a second liquid coolant between a secondelectronic device of the plurality of electronic devices and the heatexchanger, the second electronic device being thermally coupled to thesecond liquid coolant such that heat is transferred from the secondelectronic device to the second liquid coolant; wherein the firstcooling circulatory arrangement and the second cooling circulatoryarrangement are thermally coupled at least via the heat exchanger, suchthat heat is transferred from the first liquid coolant to the secondliquid coolant via the heat exchanger.
 18. The method of claim 17,wherein the second cooling circulatory arrangement further comprises acooling system, wherein the circulating the second liquid coolant aroundthe second cooling circulatory arrangement comprises circulating thesecond liquid coolant between the second electronic device of theplurality of electronic devices, the heat exchanger and the coolingsystem, where heat is removed from the second liquid coolant by thecooling system.
 19. The method of claim 17, wherein the second coolingcirculatory arrangement further comprises a second liquid coolantsupply, wherein the circulating the second liquid coolant around thesecond cooling circulatory arrangement comprises receiving the secondliquid coolant from the second liquid coolant supply, circulating thesecond liquid coolant between the second electronic device of theplurality of electronic devices and the heat exchanger and to bereturned to the second liquid coolant supply.
 20. The method of claim17, wherein the heat exchanger comprises at least a first and a secondchamber separated by a thermal interface, wherein the heat exchanger isconfigured for flow of the first liquid coolant through at least thefirst chamber, and flow of the second liquid coolant through at leastthe second chamber, such that heat is transferred from the first liquidcoolant to the second liquid coolant through the thermal interface.21.-23. (canceled)
 24. The method of claim 17, wherein the secondcooling circulatory arrangement further comprises a cooling module,configured to thermally couple the second electronic device to thesecond liquid coolant.
 25. The method of claim 17, wherein the coolingmodule comprises a cold plate, the cold plate comprising: a cold platehousing, a surface of the cold plate housing being arranged to provide athermal interface for cooling the second electronic device which isthermally coupled thereto; and at least one channel within the coldplate housing and proximate to the surface of the cold plate housing,the at least one channel arranged for the second liquid coolant to flowtherethough such that heat received from the second electronic devicethrough the surface of the cold plate housing is transferred to thesecond liquid coolant.